[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:07:03 dist-test-slave-41cl krb5kdc[5922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:07:03 dist-test-slave-41cl krb5kdc[5922](info): set up 2 sockets
Sep 29 01:07:03 dist-test-slave-41cl krb5kdc[5922](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 01:07:04.424929 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:07:04 dist-test-slave-41cl krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:07:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:07:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:07:04.597950 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:38091
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:38091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:04.921118 5935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:04.921411 5935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:04.921561 5935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:04.928584 5935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:04.928681 5935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:04.928810 5935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:04.928906 5935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:04.928987 5935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:07:04.942163 5935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:38091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:38091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:04.943696 5935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:04.945032 5935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:04.957481 5940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:04.957436 5943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:04.957458 5941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:04.958137 5935 server_base.cc:1034] running on GCE node
I20260929 01:07:04.958974 5935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:04.960505 5935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:04.961733 5935 hybrid_clock.cc:648] HybridClock initialized: now 1790644024961687 us; error 38 us; skew 500 ppm
I20260929 01:07:04.962071 5935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:04 dist-test-slave-41cl krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644024, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:04.965276 5935 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:07:04.966657 5935 webserver.cc:460] Webserver started at http://127.5.198.254:45091/ using document root <none> and password file <none>
I20260929 01:07:04.967325 5935 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:04.967460 5935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:04.967748 5935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:04.970657 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "86f8eb00687a4871801447d2dcd3cbff"
format_stamp: "Formatted at 2026-09-29 01:07:04 on dist-test-slave-41cl"
server_key: "a4ad23d9ed4b0ad1c7861336468668cc"
server_key_iv: "7442f3033ca104a8616c75e8959aa386"
server_key_version: "encryptionkey@0"
I20260929 01:07:04.971465 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "86f8eb00687a4871801447d2dcd3cbff"
format_stamp: "Formatted at 2026-09-29 01:07:04 on dist-test-slave-41cl"
server_key: "a4ad23d9ed4b0ad1c7861336468668cc"
server_key_iv: "7442f3033ca104a8616c75e8959aa386"
server_key_version: "encryptionkey@0"
I20260929 01:07:04.976686 5935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 01:07:04.980456 5950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:04.981812 5935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:07:04.982023 5935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "86f8eb00687a4871801447d2dcd3cbff"
format_stamp: "Formatted at 2026-09-29 01:07:04 on dist-test-slave-41cl"
server_key: "a4ad23d9ed4b0ad1c7861336468668cc"
server_key_iv: "7442f3033ca104a8616c75e8959aa386"
server_key_version: "encryptionkey@0"
I20260929 01:07:04.982213 5935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:04.996820 5935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:05.000798 5935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:05.001070 5935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:05.020599 5935 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:38091
I20260929 01:07:05.020605 6001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:38091 every 8 connection(s)
I20260929 01:07:05.021924 5935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:07:05.024343 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 5935
I20260929 01:07:05.024662 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:07:05.025642 6002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:05.026479 5915 external_mini_cluster.cc:1428] Setting key 8e8709f3c76120fbedac391c6cac42e6
I20260929 01:07:05.036109 6002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff: Bootstrap starting.
I20260929 01:07:05.038910 6002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:05.039844 6002 log.cc:824] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:05.043321 6002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff: No bootstrap required, opened a new log
I20260929 01:07:05.050348 6002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f8eb00687a4871801447d2dcd3cbff" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38091 } }
I20260929 01:07:05.050706 6002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:05.050806 6002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f8eb00687a4871801447d2dcd3cbff, State: Initialized, Role: FOLLOWER
I20260929 01:07:05.051789 6002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f8eb00687a4871801447d2dcd3cbff" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38091 } }
I20260929 01:07:05.052028 6002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:07:05.052163 6002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:07:05.052335 6002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [term 0 FOLLOWER]: Advancing to term 1
Sep 29 01:07:05 dist-test-slave-41cl krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:05.054867 6002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f8eb00687a4871801447d2dcd3cbff" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38091 } }
I20260929 01:07:05.055465 6002 leader_election.cc:302] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86f8eb00687a4871801447d2dcd3cbff; no voters:
I20260929 01:07:05.056169 6002 leader_election.cc:288] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:07:05.056308 6007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:05.056972 6007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [term 1 LEADER]: Becoming Leader. State: Replica: 86f8eb00687a4871801447d2dcd3cbff, State: Running, Role: LEADER
I20260929 01:07:05.057785 6007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f8eb00687a4871801447d2dcd3cbff" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38091 } }
I20260929 01:07:05.059401 6002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:05.060606 6008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86f8eb00687a4871801447d2dcd3cbff. Latest consensus state: current_term: 1 leader_uuid: "86f8eb00687a4871801447d2dcd3cbff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f8eb00687a4871801447d2dcd3cbff" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38091 } } }
I20260929 01:07:05.061605 6008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:05.061568 6009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86f8eb00687a4871801447d2dcd3cbff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f8eb00687a4871801447d2dcd3cbff" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38091 } } }
I20260929 01:07:05.062071 6009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:05.069366 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:05.030109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.5.198.254:38091)
0929 01:07:05.030978 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:05.030993 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:05.033926 (+ 2933us) server_negotiation.cc:407] Connection header received
0929 01:07:05.042449 (+ 8523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:05.042469 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:05.042811 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:05.043163 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:05.045968 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:05.047283 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:05.048955 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:05.049638 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:05.049822 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:05.056038 (+ 6216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:05.056051 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:05.056071 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:05.056134 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:05.061694 (+ 5560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:05.063070 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:05.063082 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:05.063096 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:05.063431 (+ 335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:05.064126 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:05.064135 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:05.064141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:05.065066 (+ 925us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:05.065436 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:05.066593 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0929 01:07:05.067127 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":394,"threads_started":1}
I20260929 01:07:05.073698 6013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:07:05.080253 6013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:07:05.089355 6013 catalog_manager.cc:1347] Generated new cluster ID: a671b796adc7406789a6fe1a86e997f3
I20260929 01:07:05.089478 6013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:07:05.110085 6013 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:07:05.111507 6013 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:07:05.117949 6013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86f8eb00687a4871801447d2dcd3cbff: Generated new TSK 0
I20260929 01:07:05.118815 6013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:07:05.160537 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:38091
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:05.486130 6028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:05.486435 6028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:05.486557 6028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:05.493861 6028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:05.493966 6028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:05.494087 6028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:07:05.503502 6028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:38091
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:05.504834 6028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:05.506069 6028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:05.518852 6033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:05.518959 6036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:05.519025 6034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:05.519853 6028 server_base.cc:1034] running on GCE node
I20260929 01:07:05.520414 6028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:05.521339 6028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:05.522542 6028 hybrid_clock.cc:648] HybridClock initialized: now 1790644025522483 us; error 51 us; skew 500 ppm
I20260929 01:07:05.522876 6028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:05 dist-test-slave-41cl krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644025, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:05.526489 6028 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:07:05.527861 6028 webserver.cc:460] Webserver started at http://127.5.198.193:45193/ using document root <none> and password file <none>
I20260929 01:07:05.528571 6028 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:05.528673 6028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:05.528935 6028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:05.531639 6028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "0d8124cdd6d34dae9e3b5e63f0e9f077"
format_stamp: "Formatted at 2026-09-29 01:07:05 on dist-test-slave-41cl"
server_key: "20991d4fcc49cc9e53c467baa58e4d5e"
server_key_iv: "e0d85c271ad7048a8befb6cf7402ff01"
server_key_version: "encryptionkey@0"
I20260929 01:07:05.532343 6028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "0d8124cdd6d34dae9e3b5e63f0e9f077"
format_stamp: "Formatted at 2026-09-29 01:07:05 on dist-test-slave-41cl"
server_key: "20991d4fcc49cc9e53c467baa58e4d5e"
server_key_iv: "e0d85c271ad7048a8befb6cf7402ff01"
server_key_version: "encryptionkey@0"
I20260929 01:07:05.537276 6028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 01:07:05.540567 6043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:05.541873 6028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:07:05.542080 6028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "0d8124cdd6d34dae9e3b5e63f0e9f077"
format_stamp: "Formatted at 2026-09-29 01:07:05 on dist-test-slave-41cl"
server_key: "20991d4fcc49cc9e53c467baa58e4d5e"
server_key_iv: "e0d85c271ad7048a8befb6cf7402ff01"
server_key_version: "encryptionkey@0"
I20260929 01:07:05.542240 6028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:05.557585 6028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:05.562218 6028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:05.562434 6028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:05.563259 6028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:05.564666 6028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:05.564736 6028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:05.564817 6028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:05.564862 6028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:05.605114 6028 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:34317
I20260929 01:07:05.605160 6155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:34317 every 8 connection(s)
I20260929 01:07:05.606552 6028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:07:05.616820 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 6028
I20260929 01:07:05.617064 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:07:05.617830 5915 external_mini_cluster.cc:1428] Setting key 0ab33765e663e6b479ee4d908fa46774
Sep 29 01:07:05 dist-test-slave-41cl krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644025, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:05.628305 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:05.609235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:45337 (local address 127.5.198.254:38091)
0929 01:07:05.609696 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:05.609700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:05.610661 (+ 961us) server_negotiation.cc:407] Connection header received
0929 01:07:05.612413 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:05.612417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:05.612528 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:05.612624 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:05.614116 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:05.614823 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:05.616857 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:05.617636 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:05.617788 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:05.621747 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:05.621756 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:05.621764 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:05.621809 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:05.625012 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:05.625828 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:05.625836 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:05.625842 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:05.625989 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:05.626391 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:05.626398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:05.626405 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:05.626782 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:05.626935 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:05.627594 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 01:07:05.627837 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":242,"threads_started":1}
I20260929 01:07:05.629015 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:05.609811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38091 (local address 127.5.198.193:45337)
0929 01:07:05.610506 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:05.610542 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:05.612167 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:05.612828 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:05.612837 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:05.613352 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:05.613923 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:05.613940 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:05.615376 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:05.615381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:05.616706 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:05.616717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:05.617843 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:05.617848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:05.618137 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:05.618800 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:05.618869 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:05.621521 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:05.625146 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:05.625160 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:05.625178 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:05.625712 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:05.626100 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:05.626103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:05.626105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:05.626284 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:05.626907 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:05.626913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:05.627221 (+ 308us) client_negotiation.cc:769] Sending connection context
0929 01:07:05.627456 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 01:07:05.627892 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":259,"threads_started":1}
I20260929 01:07:05.630589 6156 heartbeater.cc:347] Connected to a master server at 127.5.198.254:38091
I20260929 01:07:05.630901 6156 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:05.631630 6156 heartbeater.cc:511] Master 127.5.198.254:38091 requested a full tablet report, sending...
I20260929 01:07:05.633651 5967 ts_manager.cc:194] Registered new tserver with Master: 0d8124cdd6d34dae9e3b5e63f0e9f077 (127.5.198.193:34317)
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
I20260929 01:07:05.635076 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:45337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:07:05.660386 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:38091
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:05.982685 6162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:05.982993 6162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:05.983166 6162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:05.990878 6162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:05.991017 6162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:05.991171 6162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:07:06.000331 6162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:38091
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:06.001509 6162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:06.002779 6162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:06.016017 6167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:06.015986 6168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:06.015985 6170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:06.016817 6162 server_base.cc:1034] running on GCE node
I20260929 01:07:06.017395 6162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:06.018301 6162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:06.019742 6162 hybrid_clock.cc:648] HybridClock initialized: now 1790644026019675 us; error 53 us; skew 500 ppm
I20260929 01:07:06.020143 6162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:06 dist-test-slave-41cl krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644026, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:06.023711 6162 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:07:06.025238 6162 webserver.cc:460] Webserver started at http://127.5.198.194:46457/ using document root <none> and password file <none>
I20260929 01:07:06.025892 6162 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:06.026012 6162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:06.026310 6162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:06.029192 6162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "1f5a7aa7346b44fbacc12f9a65af451b"
format_stamp: "Formatted at 2026-09-29 01:07:06 on dist-test-slave-41cl"
server_key: "31d0087300311f5051dad4e826c5e0c0"
server_key_iv: "7c3416686933f44dd14a646389ea1e27"
server_key_version: "encryptionkey@0"
I20260929 01:07:06.029943 6162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "1f5a7aa7346b44fbacc12f9a65af451b"
format_stamp: "Formatted at 2026-09-29 01:07:06 on dist-test-slave-41cl"
server_key: "31d0087300311f5051dad4e826c5e0c0"
server_key_iv: "7c3416686933f44dd14a646389ea1e27"
server_key_version: "encryptionkey@0"
I20260929 01:07:06.034964 6162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:07:06.038267 6177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:06.039521 6162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:07:06.039680 6162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "1f5a7aa7346b44fbacc12f9a65af451b"
format_stamp: "Formatted at 2026-09-29 01:07:06 on dist-test-slave-41cl"
server_key: "31d0087300311f5051dad4e826c5e0c0"
server_key_iv: "7c3416686933f44dd14a646389ea1e27"
server_key_version: "encryptionkey@0"
I20260929 01:07:06.039928 6162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:06.063551 6162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:06.067911 6162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:06.068154 6162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:06.069106 6162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:06.070446 6162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:06.070518 6162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:06.070604 6162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:06.070647 6162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:06.105114 6162 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:37257
I20260929 01:07:06.105201 6289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:37257 every 8 connection(s)
I20260929 01:07:06.106426 6162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:07:06.116111 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 6162
I20260929 01:07:06.116298 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:07:06.116780 5915 external_mini_cluster.cc:1428] Setting key 1bfa22592a1b357a7bf0fec20cefcaea
Sep 29 01:07:06 dist-test-slave-41cl krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644026, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:06.128296 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:06.109434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:56123 (local address 127.5.198.254:38091)
0929 01:07:06.109636 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:06.109641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:06.110881 (+ 1240us) server_negotiation.cc:407] Connection header received
0929 01:07:06.112720 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:06.112727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:06.112817 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:06.112923 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:06.114540 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.115243 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:06.116984 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.117453 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:06.117575 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:06.121550 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:06.121559 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:06.121568 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:06.121611 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:06.125069 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:06.125982 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:06.125985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:06.125988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:06.126109 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:06.126605 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:06.126609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:06.126611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:06.126854 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:06.126970 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:06.127892 (+ 922us) server_negotiation.cc:299] Negotiation successful
0929 01:07:06.128056 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 01:07:06.129932 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:06.109931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38091 (local address 127.5.198.194:56123)
0929 01:07:06.110715 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:06.110765 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:06.112415 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:06.113148 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:06.113156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:06.113679 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:06.114332 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:06.114350 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.115385 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:06.115391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:06.116782 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:06.116794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.117684 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:06.117691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:06.117953 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:06.118760 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:06.118780 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:06.121341 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:06.125187 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:06.125199 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:06.125218 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:06.125841 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:06.126224 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:06.126231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:06.126238 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:06.126464 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:06.127004 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:06.127023 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:06.127454 (+ 431us) client_negotiation.cc:769] Sending connection context
0929 01:07:06.127746 (+ 292us) client_negotiation.cc:240] Negotiation successful
0929 01:07:06.128352 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":360,"threads_started":1}
I20260929 01:07:06.131431 6290 heartbeater.cc:347] Connected to a master server at 127.5.198.254:38091
I20260929 01:07:06.131770 6290 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:06.132678 6290 heartbeater.cc:511] Master 127.5.198.254:38091 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
I20260929 01:07:06.134326 5967 ts_manager.cc:194] Registered new tserver with Master: 1f5a7aa7346b44fbacc12f9a65af451b (127.5.198.194:37257)
I20260929 01:07:06.135182 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:56123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:07:06.160778 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:38091
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:06.482290 6295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:06.482609 6295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:06.482775 6295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:06.489852 6295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:06.489991 6295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:06.490170 6295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:07:06.499684 6295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:38091
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:06.500970 6295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:06.502259 6295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:06.516274 6300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:06.516245 6301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:06.516731 6303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:06.517234 6295 server_base.cc:1034] running on GCE node
I20260929 01:07:06.517802 6295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:06.518723 6295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:06.519964 6295 hybrid_clock.cc:648] HybridClock initialized: now 1790644026519904 us; error 35 us; skew 500 ppm
I20260929 01:07:06.520314 6295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:06 dist-test-slave-41cl krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644026, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:06.523435 6295 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:07:06.524757 6295 webserver.cc:460] Webserver started at http://127.5.198.195:38231/ using document root <none> and password file <none>
I20260929 01:07:06.525368 6295 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:06.525470 6295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:06.525714 6295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:06.528357 6295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "258cb9928ddc4453864d3e9fd3dba825"
format_stamp: "Formatted at 2026-09-29 01:07:06 on dist-test-slave-41cl"
server_key: "799485fbd1123c25adf1776a60ee8723"
server_key_iv: "ac5faf3d9f114836487351768387e4bc"
server_key_version: "encryptionkey@0"
I20260929 01:07:06.529232 6295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "258cb9928ddc4453864d3e9fd3dba825"
format_stamp: "Formatted at 2026-09-29 01:07:06 on dist-test-slave-41cl"
server_key: "799485fbd1123c25adf1776a60ee8723"
server_key_iv: "ac5faf3d9f114836487351768387e4bc"
server_key_version: "encryptionkey@0"
I20260929 01:07:06.533999 6295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:07:06.537267 6310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:06.538630 6295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:07:06.538786 6295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "258cb9928ddc4453864d3e9fd3dba825"
format_stamp: "Formatted at 2026-09-29 01:07:06 on dist-test-slave-41cl"
server_key: "799485fbd1123c25adf1776a60ee8723"
server_key_iv: "ac5faf3d9f114836487351768387e4bc"
server_key_version: "encryptionkey@0"
I20260929 01:07:06.538951 6295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:06.586741 6295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:06.591926 6295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:06.592270 6295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:06.593410 6295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:06.594878 6295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:06.594956 6295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:06.595144 6295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:06.595201 6295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:06.633673 6295 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:41447
I20260929 01:07:06.633710 6422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:41447 every 8 connection(s)
I20260929 01:07:06.635033 6295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:07:06.636946 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 6295
I20260929 01:07:06.637156 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:07:06.637624 5915 external_mini_cluster.cc:1428] Setting key 53beafd1fb38160f87db5d404ac4ad09
I20260929 01:07:06.638055 6156 heartbeater.cc:503] Master 127.5.198.254:38091 was elected leader, sending a full tablet report...
Sep 29 01:07:06 dist-test-slave-41cl krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644026, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:06.655827 6425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:06.637739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:34233 (local address 127.5.198.254:38091)
0929 01:07:06.638858 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:06.638864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:06.639896 (+ 1032us) server_negotiation.cc:407] Connection header received
0929 01:07:06.641370 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:06.641376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:06.641466 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:06.641580 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:06.643151 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.644156 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:06.645845 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.646320 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:06.646424 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:06.650017 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:06.650023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:06.650026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:06.650056 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:06.652855 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:06.653734 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:06.653739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:06.653742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:06.653881 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:06.654274 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:06.654277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:06.654280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:06.654554 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:06.654659 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:06.655396 (+ 737us) server_negotiation.cc:299] Negotiation successful
0929 01:07:06.655570 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993,"thread_start_us":1209,"threads_started":1}
I20260929 01:07:06.657125 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:06.638373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38091 (local address 127.5.198.195:34233)
0929 01:07:06.639193 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:06.639244 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:06.641105 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:06.641843 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:06.641851 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:06.642339 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:06.642971 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:06.643007 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.644289 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:06.644293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:06.645693 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:06.645705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.646519 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:06.646524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:06.646713 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:06.647301 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:06.647322 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:06.649831 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:06.653022 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:06.653031 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:06.653047 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:06.653604 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:06.653992 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:06.653995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:06.653998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:06.654165 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:06.654658 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:06.654664 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:06.655022 (+ 358us) client_negotiation.cc:769] Sending connection context
0929 01:07:06.655249 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 01:07:06.655789 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":369,"threads_started":1}
I20260929 01:07:06.658350 6423 heartbeater.cc:347] Connected to a master server at 127.5.198.254:38091
I20260929 01:07:06.658632 6423 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:06.659318 6423 heartbeater.cc:511] Master 127.5.198.254:38091 requested a full tablet report, sending...
I20260929 01:07:06.660893 5967 ts_manager.cc:194] Registered new tserver with Master: 258cb9928ddc4453864d3e9fd3dba825 (127.5.198.195:41447)
I20260929 01:07:06.661729 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:34233
I20260929 01:07:06.666239 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 01:07:06 dist-test-slave-41cl krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644026, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 01:07:06 dist-test-slave-41cl krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644026, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:07:06.711512 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:06.695844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.5.198.193:34317)
0929 01:07:06.696444 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:06.696451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:06.696550 (+ 99us) server_negotiation.cc:407] Connection header received
0929 01:07:06.696621 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:06.696628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:06.696849 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:06.696973 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:06.698807 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.699837 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:06.701159 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.701781 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:06.701891 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:06.705761 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:06.705772 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:06.705786 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:06.705817 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:06.708718 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:06.709369 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:06.709378 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:06.709381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:06.709567 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:06.709980 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:06.709984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:06.709987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:06.710362 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:06.710582 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:06.710912 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 01:07:06.711179 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":308,"threads_started":1}
W20260929 01:07:06.712941 6071 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43026
Sep 29 01:07:06 dist-test-slave-41cl krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 01:07:06 dist-test-slave-41cl krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:07:06.756510 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:06.742679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.5.198.193:34317)
0929 01:07:06.742967 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:06.742975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:06.743215 (+ 240us) server_negotiation.cc:407] Connection header received
0929 01:07:06.743381 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:06.743385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:06.743457 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:06.743553 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:06.745039 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.745764 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:06.747065 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.747509 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:06.747613 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:06.751790 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:06.751795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:06.751798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:06.751829 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:06.754409 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:06.754992 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:06.754996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:06.754998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:06.755110 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:06.755414 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:06.755418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:06.755422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:06.755641 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:06.755805 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:06.756070 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 01:07:06.756221 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:07:06.758023 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 6028
I20260929 01:07:06.770160 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:06.758993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.5.198.193:34317)
0929 01:07:06.759220 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:06.759226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:06.759407 (+ 181us) server_negotiation.cc:407] Connection header received
0929 01:07:06.759530 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:06.759533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:06.759583 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:06.759689 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:06.760932 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.761598 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:06.762720 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.763113 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:06.763202 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:06.765230 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:06.765234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:06.765238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:06.765264 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:06.767490 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:06.768031 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:06.768034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:06.768036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:06.768153 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:06.768946 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:06.768950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:06.768952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:06.769171 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:06.769338 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:06.769635 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 01:07:06.769843 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 01:07:06.770898 6151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:06.852809 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 6028
I20260929 01:07:06.869758 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 6162
Sep 29 01:07:06 dist-test-slave-41cl krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:07:06.886950 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:06.870670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52346 (local address 127.5.198.194:37257)
0929 01:07:06.871266 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:06.871274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:06.871302 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:07:06.871377 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:06.871385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:06.871617 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:06.871818 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:06.873313 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.874546 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:06.875992 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.876813 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:06.876947 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:06.881205 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:06.881216 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:06.881230 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:06.881270 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:06.884021 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:06.884741 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:06.884748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:06.884750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:06.884887 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:06.885298 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:06.885301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:06.885303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:06.885681 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:06.885995 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:06.886302 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 01:07:06.886578 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":323,"threads_started":1}
I20260929 01:07:06.888334 6285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:06.969518 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 6162
I20260929 01:07:06.994689 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 6295
Sep 29 01:07:07 dist-test-slave-41cl krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:07:07.010994 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:06.995518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.5.198.195:41447)
0929 01:07:06.996109 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:06.996116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:06.996144 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:07:06.996200 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:06.996206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:06.996415 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:06.996641 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:06.997910 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:06.998949 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:07.000227 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:07.000857 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:07.000995 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:07.005139 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:07.005150 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:07.005163 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:07.005194 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:07.007990 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:07.009076 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:07.009084 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:07.009086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:07.009211 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:07.009599 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:07.009603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:07.009605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:07.009928 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:07.010172 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:07.010393 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 01:07:07.010593 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":267,"threads_started":1}
I20260929 01:07:07.012251 6418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:07.098836 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 6295
I20260929 01:07:07.115105 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 5935
I20260929 01:07:07.116204 5997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:07.218189 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 5935
2026-09-29T01:07:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3891 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:07:07 dist-test-slave-41cl krb5kdc[6445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:07:07 dist-test-slave-41cl krb5kdc[6445](info): set up 2 sockets
Sep 29 01:07:07 dist-test-slave-41cl krb5kdc[6445](info): commencing operation
krb5kdc: starting...
W20260929 01:07:08.304657 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:07:08 dist-test-slave-41cl krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:07:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:07:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:07:08.484083 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:42089
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:42089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:08.797703 6458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:08.797979 6458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:08.798175 6458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:08.805367 6458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:08.805464 6458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:08.805560 6458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:08.805653 6458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:08.805729 6458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:07:08.815411 6458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:42089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:42089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:08.816768 6458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:08.818068 6458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:08.829290 6463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:08.829329 6464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:08.829377 6466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:08.830500 6458 server_base.cc:1034] running on GCE node
I20260929 01:07:08.831282 6458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:08.832645 6458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:08.833837 6458 hybrid_clock.cc:648] HybridClock initialized: now 1790644028833753 us; error 75 us; skew 500 ppm
I20260929 01:07:08.834179 6458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:08 dist-test-slave-41cl krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644028, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:08.844590 6458 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:07:08.846078 6458 webserver.cc:460] Webserver started at http://127.5.198.254:34763/ using document root <none> and password file <none>
I20260929 01:07:08.846730 6458 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:08.846827 6458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:08.847077 6458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:08.849946 6458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "7372f9537eb0402b93b473114795e785"
format_stamp: "Formatted at 2026-09-29 01:07:08 on dist-test-slave-41cl"
server_key: "7680b2df8e01323224f2295a1cb16156"
server_key_iv: "d8868d40d3917ccdebe9e28065539f22"
server_key_version: "encryptionkey@0"
I20260929 01:07:08.850620 6458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "7372f9537eb0402b93b473114795e785"
format_stamp: "Formatted at 2026-09-29 01:07:08 on dist-test-slave-41cl"
server_key: "7680b2df8e01323224f2295a1cb16156"
server_key_iv: "d8868d40d3917ccdebe9e28065539f22"
server_key_version: "encryptionkey@0"
I20260929 01:07:08.855618 6458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 01:07:08.859059 6473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:08.860389 6458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:07:08.860605 6458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "7372f9537eb0402b93b473114795e785"
format_stamp: "Formatted at 2026-09-29 01:07:08 on dist-test-slave-41cl"
server_key: "7680b2df8e01323224f2295a1cb16156"
server_key_iv: "d8868d40d3917ccdebe9e28065539f22"
server_key_version: "encryptionkey@0"
I20260929 01:07:08.860775 6458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:08.881803 6458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:08.885869 6458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:08.886087 6458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:08.903792 6458 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:42089
I20260929 01:07:08.903811 6524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:42089 every 8 connection(s)
I20260929 01:07:08.905112 6458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:07:08.908670 6525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:08.909847 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 6458
I20260929 01:07:08.910077 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:07:08.910892 5915 external_mini_cluster.cc:1428] Setting key 5caa98f5a42b18180ed80370369b4b7c
I20260929 01:07:08.921564 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785: Bootstrap starting.
Sep 29 01:07:08 dist-test-slave-41cl krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:08.924908 6525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:08.925992 6525 log.cc:824] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:08.929111 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785: No bootstrap required, opened a new log
I20260929 01:07:08.932667 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:08.913107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.5.198.254:42089)
0929 01:07:08.913857 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:08.913873 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:08.913932 (+ 59us) server_negotiation.cc:407] Connection header received
0929 01:07:08.915476 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:08.915499 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:08.915888 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:08.916270 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:08.917743 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:08.919043 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:08.920284 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:08.921051 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:08.921263 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:08.925398 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:08.925406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:08.925421 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:08.925459 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:08.928688 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:08.929385 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:08.929392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:08.929398 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:08.929554 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:08.929912 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:08.929921 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:08.929923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:08.930403 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:08.930738 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:08.931216 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 01:07:08.931617 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":326,"threads_started":1}
I20260929 01:07:08.937407 6525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7372f9537eb0402b93b473114795e785" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42089 } }
I20260929 01:07:08.937774 6525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:08.937883 6525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7372f9537eb0402b93b473114795e785, State: Initialized, Role: FOLLOWER
I20260929 01:07:08.938437 6525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7372f9537eb0402b93b473114795e785" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42089 } }
I20260929 01:07:08.938675 6525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:07:08.938807 6525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:07:08.938970 6525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:08.941431 6525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7372f9537eb0402b93b473114795e785" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42089 } }
I20260929 01:07:08.941915 6525 leader_election.cc:302] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7372f9537eb0402b93b473114795e785; no voters:
I20260929 01:07:08.942500 6525 leader_election.cc:288] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:07:08.942664 6530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:08.943357 6530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [term 1 LEADER]: Becoming Leader. State: Replica: 7372f9537eb0402b93b473114795e785, State: Running, Role: LEADER
I20260929 01:07:08.944061 6530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7372f9537eb0402b93b473114795e785" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42089 } }
I20260929 01:07:08.944917 6525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:08.946864 6532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7372f9537eb0402b93b473114795e785. Latest consensus state: current_term: 1 leader_uuid: "7372f9537eb0402b93b473114795e785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7372f9537eb0402b93b473114795e785" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42089 } } }
I20260929 01:07:08.947329 6532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:08.948161 6531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7372f9537eb0402b93b473114795e785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7372f9537eb0402b93b473114795e785" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42089 } } }
I20260929 01:07:08.948664 6531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:08.953469 6535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:07:08.958623 6535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:07:08.967816 6535 catalog_manager.cc:1347] Generated new cluster ID: 67e62c853b3b43a5bbab83a9987012ac
I20260929 01:07:08.967933 6535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:07:08.992586 6535 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:07:08.993993 6535 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:07:09.001330 6535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7372f9537eb0402b93b473114795e785: Generated new TSK 0
I20260929 01:07:09.002239 6535 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:07:09.054461 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42089
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:09.367554 6551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:09.367818 6551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:09.367981 6551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:09.375175 6551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:09.375281 6551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:09.375452 6551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:07:09.385104 6551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42089
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:09.386365 6551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:09.387697 6551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:09.400370 6557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:09.400483 6556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:09.400569 6559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:09.401113 6551 server_base.cc:1034] running on GCE node
I20260929 01:07:09.401678 6551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:09.402552 6551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:09.403764 6551 hybrid_clock.cc:648] HybridClock initialized: now 1790644029403711 us; error 49 us; skew 500 ppm
I20260929 01:07:09.404088 6551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:09 dist-test-slave-41cl krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644029, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:09.407164 6551 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:07:09.408635 6551 webserver.cc:460] Webserver started at http://127.5.198.193:41425/ using document root <none> and password file <none>
I20260929 01:07:09.409288 6551 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:09.409403 6551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:09.409689 6551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:09.412379 6551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "df35365ecb2d431090fb98cade9df0a8"
format_stamp: "Formatted at 2026-09-29 01:07:09 on dist-test-slave-41cl"
server_key: "ca7fcc9254a55fc57feb9a09df8c043f"
server_key_iv: "a461d9e12365c52faf2f33ae7f9e39dd"
server_key_version: "encryptionkey@0"
I20260929 01:07:09.413166 6551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "df35365ecb2d431090fb98cade9df0a8"
format_stamp: "Formatted at 2026-09-29 01:07:09 on dist-test-slave-41cl"
server_key: "ca7fcc9254a55fc57feb9a09df8c043f"
server_key_iv: "a461d9e12365c52faf2f33ae7f9e39dd"
server_key_version: "encryptionkey@0"
I20260929 01:07:09.417964 6551 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:07:09.421306 6566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:09.422536 6551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:07:09.422696 6551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "df35365ecb2d431090fb98cade9df0a8"
format_stamp: "Formatted at 2026-09-29 01:07:09 on dist-test-slave-41cl"
server_key: "ca7fcc9254a55fc57feb9a09df8c043f"
server_key_iv: "a461d9e12365c52faf2f33ae7f9e39dd"
server_key_version: "encryptionkey@0"
I20260929 01:07:09.422881 6551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:09.436296 6551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:09.440310 6551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:09.440634 6551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:09.441514 6551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:09.442736 6551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:09.442832 6551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:09.442971 6551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:09.443043 6551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:09.475322 6551 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:42527
I20260929 01:07:09.475365 6678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:42527 every 8 connection(s)
I20260929 01:07:09.476625 6551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:07:09.480110 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 6551
I20260929 01:07:09.480329 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:07:09.481170 5915 external_mini_cluster.cc:1428] Setting key e055e6b87e8f75ef55c1b023f5a62e15
Sep 29 01:07:09 dist-test-slave-41cl krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644029, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
I20260929 01:07:09.499567 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:09.479041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:38399 (local address 127.5.198.254:42089)
0929 01:07:09.479465 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:09.479470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:09.480506 (+ 1036us) server_negotiation.cc:407] Connection header received
0929 01:07:09.482633 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:09.482639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:09.482706 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:09.482836 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:09.484921 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:09.485629 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:09.488039 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:09.488773 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:09.488922 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:09.492494 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:09.492504 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:09.492512 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:09.492556 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:09.496150 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:09.496968 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:09.496976 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:09.496981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:09.497123 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:09.497602 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:09.497609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:09.497613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:09.497989 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:09.498147 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:09.498835 (+ 688us) server_negotiation.cc:299] Negotiation successful
0929 01:07:09.499065 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":228,"threads_started":1}
I20260929 01:07:09.500416 6682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:09.479463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:42089 (local address 127.5.198.193:38399)
0929 01:07:09.480265 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:09.480324 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:09.482383 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:09.483136 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:09.483148 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:09.483872 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:09.484731 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:09.484759 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:09.485785 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:09.485794 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:09.487887 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:09.487898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:09.489009 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:09.489014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:09.489212 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:09.489901 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:09.489923 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:09.492244 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:09.496338 (+ 4094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:09.496345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:09.496358 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:09.496842 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:09.497239 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:09.497242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:09.497244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:09.497475 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:09.498122 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:09.498127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:09.498466 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 01:07:09.498689 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 01:07:09.499067 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":278,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
I20260929 01:07:09.502206 6679 heartbeater.cc:347] Connected to a master server at 127.5.198.254:42089
I20260929 01:07:09.502532 6679 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:09.503234 6679 heartbeater.cc:511] Master 127.5.198.254:42089 requested a full tablet report, sending...
I20260929 01:07:09.505519 6490 ts_manager.cc:194] Registered new tserver with Master: df35365ecb2d431090fb98cade9df0a8 (127.5.198.193:42527)
I20260929 01:07:09.507727 6490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:38399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:07:09.524055 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42089
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:09.837306 6685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:09.837616 6685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:09.837738 6685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:09.844728 6685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:09.844825 6685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:09.844949 6685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:07:09.854668 6685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42089
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:09.856324 6685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:09.858282 6685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:09.871816 6691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:09.871840 6690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:09.872332 6693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:09.872713 6685 server_base.cc:1034] running on GCE node
I20260929 01:07:09.873303 6685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:09.874209 6685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:09.875428 6685 hybrid_clock.cc:648] HybridClock initialized: now 1790644029875384 us; error 60 us; skew 500 ppm
I20260929 01:07:09.875716 6685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:09 dist-test-slave-41cl krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644029, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:09.878728 6685 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:07:09.880023 6685 webserver.cc:460] Webserver started at http://127.5.198.194:35291/ using document root <none> and password file <none>
I20260929 01:07:09.880745 6685 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:09.880874 6685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:09.881176 6685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:09.883877 6685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c"
format_stamp: "Formatted at 2026-09-29 01:07:09 on dist-test-slave-41cl"
server_key: "386ac51f7221e0dbbc83f2643dc3c14d"
server_key_iv: "94b04945ddf2b7694b1a77c5e9e8f3e5"
server_key_version: "encryptionkey@0"
I20260929 01:07:09.884680 6685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c"
format_stamp: "Formatted at 2026-09-29 01:07:09 on dist-test-slave-41cl"
server_key: "386ac51f7221e0dbbc83f2643dc3c14d"
server_key_iv: "94b04945ddf2b7694b1a77c5e9e8f3e5"
server_key_version: "encryptionkey@0"
I20260929 01:07:09.889598 6685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:07:09.892951 6700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:09.894263 6685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:07:09.894434 6685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c"
format_stamp: "Formatted at 2026-09-29 01:07:09 on dist-test-slave-41cl"
server_key: "386ac51f7221e0dbbc83f2643dc3c14d"
server_key_iv: "94b04945ddf2b7694b1a77c5e9e8f3e5"
server_key_version: "encryptionkey@0"
I20260929 01:07:09.894637 6685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:09.922719 6685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:09.927081 6685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:09.927386 6685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:09.928318 6685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:09.929733 6685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:09.929802 6685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:09.929883 6685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:09.929926 6685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:09.964735 6685 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:46491
I20260929 01:07:09.964764 6812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:46491 every 8 connection(s)
I20260929 01:07:09.966089 6685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:07:09.969931 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 6685
I20260929 01:07:09.970129 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:07:09.970631 5915 external_mini_cluster.cc:1428] Setting key 1240ef35580bcaf196a9d84e17e9eb67
Sep 29 01:07:09 dist-test-slave-41cl krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644029, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
I20260929 01:07:09.989364 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:09.968725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:49379 (local address 127.5.198.254:42089)
0929 01:07:09.968893 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:09.968899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:09.970096 (+ 1197us) server_negotiation.cc:407] Connection header received
0929 01:07:09.972417 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:09.972425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:09.972526 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:09.972626 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:09.974765 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:09.975464 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:09.977902 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:09.978618 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:09.978774 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:09.982104 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:09.982113 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:09.982121 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:09.982163 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:09.985957 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:09.986944 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:09.986953 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:09.986958 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:09.987145 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:09.987597 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:09.987605 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:09.987611 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:09.988029 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:09.988198 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:09.988954 (+ 756us) server_negotiation.cc:299] Negotiation successful
0929 01:07:09.989120 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:07:09.990381 6815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:09.969204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:42089 (local address 127.5.198.194:49379)
0929 01:07:09.969939 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:09.969978 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:09.972030 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:09.972938 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:09.972951 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:09.973732 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:09.974581 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:09.974609 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:09.975621 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:09.975629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:09.977768 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:09.977780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:09.978817 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:09.978821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:09.979061 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:09.979741 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:09.979763 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:09.981878 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:09.986216 (+ 4338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:09.986224 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:09.986238 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:09.986683 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:09.987268 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:09.987272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:09.987274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:09.987467 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:09.988181 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:09.988187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:09.988586 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 01:07:09.988830 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 01:07:09.989234 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":295,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
I20260929 01:07:09.992110 6813 heartbeater.cc:347] Connected to a master server at 127.5.198.254:42089
I20260929 01:07:09.992496 6813 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:09.993465 6813 heartbeater.cc:511] Master 127.5.198.254:42089 requested a full tablet report, sending...
I20260929 01:07:09.995158 6490 ts_manager.cc:194] Registered new tserver with Master: 92bb223cbcbe4d5f8b9543d8b1995b2c (127.5.198.194:46491)
I20260929 01:07:09.996172 6490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:49379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:07:10.016264 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42089
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:10.327533 6818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:10.327852 6818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:10.327952 6818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:10.335145 6818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:10.335253 6818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:10.335367 6818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:07:10.344794 6818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42089
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:10.346140 6818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:10.347429 6818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:10.360709 6823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:10.360668 6824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:10.360775 6826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:10.361423 6818 server_base.cc:1034] running on GCE node
I20260929 01:07:10.362071 6818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:10.362998 6818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:10.364226 6818 hybrid_clock.cc:648] HybridClock initialized: now 1790644030364150 us; error 62 us; skew 500 ppm
I20260929 01:07:10.364589 6818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:10 dist-test-slave-41cl krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644030, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:10.367789 6818 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:07:10.369393 6818 webserver.cc:460] Webserver started at http://127.5.198.195:38577/ using document root <none> and password file <none>
I20260929 01:07:10.370069 6818 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:10.370188 6818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:10.370487 6818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:10.373386 6818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "5bdb606e9ba2492f94c9f32eacd6f293"
format_stamp: "Formatted at 2026-09-29 01:07:10 on dist-test-slave-41cl"
server_key: "5e4afa3a565da216bfeac53f6cf48bf0"
server_key_iv: "5f255767bed1cce4704751116f6684e4"
server_key_version: "encryptionkey@0"
I20260929 01:07:10.374145 6818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "5bdb606e9ba2492f94c9f32eacd6f293"
format_stamp: "Formatted at 2026-09-29 01:07:10 on dist-test-slave-41cl"
server_key: "5e4afa3a565da216bfeac53f6cf48bf0"
server_key_iv: "5f255767bed1cce4704751116f6684e4"
server_key_version: "encryptionkey@0"
I20260929 01:07:10.379470 6818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:07:10.382906 6833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:10.384320 6818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 01:07:10.384564 6818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "5bdb606e9ba2492f94c9f32eacd6f293"
format_stamp: "Formatted at 2026-09-29 01:07:10 on dist-test-slave-41cl"
server_key: "5e4afa3a565da216bfeac53f6cf48bf0"
server_key_iv: "5f255767bed1cce4704751116f6684e4"
server_key_version: "encryptionkey@0"
I20260929 01:07:10.384779 6818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:10.401156 6818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:10.405177 6818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:10.405444 6818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:10.406374 6818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:10.407656 6818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:10.407763 6818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:10.407902 6818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:10.407974 6818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:10.440299 6818 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:38781
I20260929 01:07:10.440340 6945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:38781 every 8 connection(s)
I20260929 01:07:10.441552 6818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:07:10.452014 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 6818
I20260929 01:07:10.452224 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:07:10.452718 5915 external_mini_cluster.cc:1428] Setting key 7460d0107c77883c95c0ef1546dea1da
Sep 29 01:07:10 dist-test-slave-41cl krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644030, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:10.462942 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.444163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:41687 (local address 127.5.198.254:42089)
0929 01:07:10.444363 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:10.444376 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:10.445646 (+ 1270us) server_negotiation.cc:407] Connection header received
0929 01:07:10.447520 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:10.447525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:10.447590 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:10.447849 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:10.449286 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.450017 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.452194 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.452723 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.452858 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.456836 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:10.456876 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:10.456879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:10.456910 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:10.460097 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:10.460814 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:10.460818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:10.460820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:10.460942 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:10.461331 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:10.461335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:10.461337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:10.461625 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:10.461741 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:10.462486 (+ 745us) server_negotiation.cc:299] Negotiation successful
0929 01:07:10.462640 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 01:07:10.463954 6948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.444756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:42089 (local address 127.5.198.195:41687)
0929 01:07:10.445498 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:10.445537 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:10.447259 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:10.448068 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:10.448076 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:10.448588 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:10.449136 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:10.449153 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.450147 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.450151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:10.452024 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:10.452040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.452992 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.453002 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:10.453317 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.454073 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:10.454103 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:10.456646 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:10.460229 (+ 3583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:10.460235 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:10.460247 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:10.460713 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:10.461030 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:10.461033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:10.461035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:10.461204 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:10.461781 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:10.461787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:10.462137 (+ 350us) client_negotiation.cc:769] Sending connection context
0929 01:07:10.462372 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 01:07:10.462750 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":312,"threads_started":1}
I20260929 01:07:10.465483 6946 heartbeater.cc:347] Connected to a master server at 127.5.198.254:42089
I20260929 01:07:10.465744 6946 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:10.466552 6946 heartbeater.cc:511] Master 127.5.198.254:42089 requested a full tablet report, sending...
I20260929 01:07:10.468017 6489 ts_manager.cc:194] Registered new tserver with Master: 5bdb606e9ba2492f94c9f32eacd6f293 (127.5.198.195:38781)
I20260929 01:07:10.468920 6489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:41687
I20260929 01:07:10.480233 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:07:10.499843 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.486951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47334 (local address 127.5.198.254:42089)
0929 01:07:10.487121 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:10.487126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:10.487649 (+ 523us) server_negotiation.cc:407] Connection header received
0929 01:07:10.487883 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:10.487886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:10.487951 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:10.488055 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:10.489682 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.490469 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.491847 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.492277 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.492408 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.494738 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:10.494743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:10.494747 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:10.494777 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:10.497291 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:10.497974 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:10.497980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:10.497988 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:10.498102 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:10.498454 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:10.498459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:10.498462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:10.498773 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:10.499021 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:10.499335 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 01:07:10.499559 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 01:07:10.510710 6679 heartbeater.cc:503] Master 127.5.198.254:42089 was elected leader, sending a full tablet report...
I20260929 01:07:10.506155 6489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:07:10.513787 6489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:07:10.539672 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.526097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.5.198.193:42527)
0929 01:07:10.527145 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:10.527156 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:10.528583 (+ 1427us) server_negotiation.cc:407] Connection header received
0929 01:07:10.529000 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:10.529011 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:10.529281 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:10.529446 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:10.530669 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.532455 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.535870 (+ 3415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.537838 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.537983 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.538287 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:10.539140 (+ 853us) server_negotiation.cc:299] Negotiation successful
0929 01:07:10.539316 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":1060,"threads_started":1}
I20260929 01:07:10.541107 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.525910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:42527 (local address 127.0.0.1:35252)
0929 01:07:10.528353 (+ 2443us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:10.528392 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:10.528726 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:10.529855 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:10.529866 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:10.529911 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:10.530417 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:10.530433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.532642 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.532650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:10.535678 (+ 3028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:10.535690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.538495 (+ 2805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.538505 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:10.538800 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.538884 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 01:07:10.540312 (+ 1428us) client_negotiation.cc:240] Negotiation successful
0929 01:07:10.540610 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2191,"threads_started":1}
I20260929 01:07:10.548635 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.526678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:38781 (local address 127.0.0.1:55888)
0929 01:07:10.541602 (+ 14924us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:10.541627 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:10.541747 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:10.542838 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:10.542843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:10.542866 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:10.543096 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:10.543107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.544944 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.544949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:10.546574 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:10.546589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.547807 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.547812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:10.548007 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.548048 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 01:07:10.548144 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 01:07:10.548276 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14771,"mutex_wait_us":8,"thread_start_us":1461,"threads_started":1}
I20260929 01:07:10.548190 6614 tablet_service.cc:1467] Processing CreateTablet for tablet c940bac338a54f6fa1e14b9836bc6698 (DEFAULT_TABLE table=test-table [id=a5af0d86b2f941989a5de8d98b835e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:10.549522 6614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c940bac338a54f6fa1e14b9836bc6698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:10.549513 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.527387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.5.198.195:38781)
0929 01:07:10.542358 (+ 14971us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:10.542365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:10.542390 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:07:10.542463 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:10.542468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:10.542651 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:10.542873 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:10.543263 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.544757 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.546765 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.547652 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.547797 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.548029 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:10.549010 (+ 981us) server_negotiation.cc:299] Negotiation successful
0929 01:07:10.549171 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14809,"thread_start_us":14654,"threads_started":1}
I20260929 01:07:10.550904 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.526522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:46491 (local address 127.0.0.1:57542)
0929 01:07:10.535609 (+ 9087us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:10.535638 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:10.535787 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:10.542962 (+ 7175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:10.542967 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:10.542995 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:10.543469 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:10.543483 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.545638 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.545643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:10.548030 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:10.548044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.550076 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.550082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:10.550262 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.550293 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 01:07:10.550388 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 01:07:10.550511 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8902,"mutex_wait_us":25,"thread_start_us":8747,"threads_started":1}
I20260929 01:07:10.551610 6962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.527179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57542 (local address 127.5.198.194:46491)
0929 01:07:10.542248 (+ 15069us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:10.542261 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:10.542299 (+ 38us) server_negotiation.cc:407] Connection header received
0929 01:07:10.542422 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:10.542434 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:10.542740 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:10.542976 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:10.543679 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.545477 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.548168 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.549605 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.549735 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.549940 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:10.551123 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0929 01:07:10.551283 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14841,"thread_start_us":15986,"threads_started":1}
I20260929 01:07:10.552918 6881 tablet_service.cc:1467] Processing CreateTablet for tablet c940bac338a54f6fa1e14b9836bc6698 (DEFAULT_TABLE table=test-table [id=a5af0d86b2f941989a5de8d98b835e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:10.554196 6881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c940bac338a54f6fa1e14b9836bc6698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:10.554749 6748 tablet_service.cc:1467] Processing CreateTablet for tablet c940bac338a54f6fa1e14b9836bc6698 (DEFAULT_TABLE table=test-table [id=a5af0d86b2f941989a5de8d98b835e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:10.556375 6748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c940bac338a54f6fa1e14b9836bc6698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:10.565233 6964 tablet_bootstrap.cc:492] T c940bac338a54f6fa1e14b9836bc6698 P 5bdb606e9ba2492f94c9f32eacd6f293: Bootstrap starting.
I20260929 01:07:10.567704 6965 tablet_bootstrap.cc:492] T c940bac338a54f6fa1e14b9836bc6698 P df35365ecb2d431090fb98cade9df0a8: Bootstrap starting.
I20260929 01:07:10.569062 6964 tablet_bootstrap.cc:654] T c940bac338a54f6fa1e14b9836bc6698 P 5bdb606e9ba2492f94c9f32eacd6f293: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:10.570274 6964 log.cc:824] T c940bac338a54f6fa1e14b9836bc6698 P 5bdb606e9ba2492f94c9f32eacd6f293: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:10.570488 6966 tablet_bootstrap.cc:492] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c: Bootstrap starting.
I20260929 01:07:10.571123 6965 tablet_bootstrap.cc:654] T c940bac338a54f6fa1e14b9836bc6698 P df35365ecb2d431090fb98cade9df0a8: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:10.572655 6965 log.cc:824] T c940bac338a54f6fa1e14b9836bc6698 P df35365ecb2d431090fb98cade9df0a8: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:10.574098 6966 tablet_bootstrap.cc:654] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:10.574119 6964 tablet_bootstrap.cc:492] T c940bac338a54f6fa1e14b9836bc6698 P 5bdb606e9ba2492f94c9f32eacd6f293: No bootstrap required, opened a new log
I20260929 01:07:10.574373 6964 ts_tablet_manager.cc:1397] T c940bac338a54f6fa1e14b9836bc6698 P 5bdb606e9ba2492f94c9f32eacd6f293: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 01:07:10.575032 6966 log.cc:824] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:10.575510 6965 tablet_bootstrap.cc:492] T c940bac338a54f6fa1e14b9836bc6698 P df35365ecb2d431090fb98cade9df0a8: No bootstrap required, opened a new log
I20260929 01:07:10.575796 6965 ts_tablet_manager.cc:1397] T c940bac338a54f6fa1e14b9836bc6698 P df35365ecb2d431090fb98cade9df0a8: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260929 01:07:10.577768 6966 tablet_bootstrap.cc:492] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c: No bootstrap required, opened a new log
I20260929 01:07:10.578011 6966 ts_tablet_manager.cc:1397] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 01:07:10.586019 6966 raft_consensus.cc:348] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df35365ecb2d431090fb98cade9df0a8" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 42527 } } peers { permanent_uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46491 } } peers { permanent_uuid: "5bdb606e9ba2492f94c9f32eacd6f293" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38781 } }
I20260929 01:07:10.586596 6966 raft_consensus.cc:374] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:10.586723 6966 raft_consensus.cc:729] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92bb223cbcbe4d5f8b9543d8b1995b2c, State: Initialized, Role: FOLLOWER
I20260929 01:07:10.586766 6964 raft_consensus.cc:348] T c940bac338a54f6fa1e14b9836bc6698 P 5bdb606e9ba2492f94c9f32eacd6f293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df35365ecb2d431090fb98cade9df0a8" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 42527 } } peers { permanent_uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46491 } } peers { permanent_uuid: "5bdb606e9ba2492f94c9f32eacd6f293" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38781 } }
I20260929 01:07:10.587337 6964 raft_consensus.cc:374] T c940bac338a54f6fa1e14b9836bc6698 P 5bdb606e9ba2492f94c9f32eacd6f293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:10.587548 6964 raft_consensus.cc:729] T c940bac338a54f6fa1e14b9836bc6698 P 5bdb606e9ba2492f94c9f32eacd6f293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bdb606e9ba2492f94c9f32eacd6f293, State: Initialized, Role: FOLLOWER
I20260929 01:07:10.587479 6966 consensus_queue.cc:260] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df35365ecb2d431090fb98cade9df0a8" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 42527 } } peers { permanent_uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46491 } } peers { permanent_uuid: "5bdb606e9ba2492f94c9f32eacd6f293" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38781 } }
I20260929 01:07:10.588732 6813 heartbeater.cc:503] Master 127.5.198.254:42089 was elected leader, sending a full tablet report...
I20260929 01:07:10.588683 6964 consensus_queue.cc:260] T c940bac338a54f6fa1e14b9836bc6698 P 5bdb606e9ba2492f94c9f32eacd6f293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df35365ecb2d431090fb98cade9df0a8" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 42527 } } peers { permanent_uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46491 } } peers { permanent_uuid: "5bdb606e9ba2492f94c9f32eacd6f293" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38781 } }
I20260929 01:07:10.588683 6965 raft_consensus.cc:348] T c940bac338a54f6fa1e14b9836bc6698 P df35365ecb2d431090fb98cade9df0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df35365ecb2d431090fb98cade9df0a8" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 42527 } } peers { permanent_uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46491 } } peers { permanent_uuid: "5bdb606e9ba2492f94c9f32eacd6f293" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38781 } }
I20260929 01:07:10.589329 6965 raft_consensus.cc:374] T c940bac338a54f6fa1e14b9836bc6698 P df35365ecb2d431090fb98cade9df0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:10.589819 6965 raft_consensus.cc:729] T c940bac338a54f6fa1e14b9836bc6698 P df35365ecb2d431090fb98cade9df0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df35365ecb2d431090fb98cade9df0a8, State: Initialized, Role: FOLLOWER
I20260929 01:07:10.590915 6965 consensus_queue.cc:260] T c940bac338a54f6fa1e14b9836bc6698 P df35365ecb2d431090fb98cade9df0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df35365ecb2d431090fb98cade9df0a8" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 42527 } } peers { permanent_uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46491 } } peers { permanent_uuid: "5bdb606e9ba2492f94c9f32eacd6f293" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38781 } }
I20260929 01:07:10.591372 6966 ts_tablet_manager.cc:1428] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 01:07:10.592734 6946 heartbeater.cc:503] Master 127.5.198.254:42089 was elected leader, sending a full tablet report...
I20260929 01:07:10.593624 6964 ts_tablet_manager.cc:1428] T c940bac338a54f6fa1e14b9836bc6698 P 5bdb606e9ba2492f94c9f32eacd6f293: Time spent starting tablet: real 0.019s user 0.012s sys 0.008s
I20260929 01:07:10.595283 6965 ts_tablet_manager.cc:1428] T c940bac338a54f6fa1e14b9836bc6698 P df35365ecb2d431090fb98cade9df0a8: Time spent starting tablet: real 0.019s user 0.015s sys 0.003s
I20260929 01:07:10.627260 6970 raft_consensus.cc:482] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:07:10.627588 6970 raft_consensus.cc:504] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df35365ecb2d431090fb98cade9df0a8" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 42527 } } peers { permanent_uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46491 } } peers { permanent_uuid: "5bdb606e9ba2492f94c9f32eacd6f293" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38781 } }
I20260929 01:07:10.629321 6970 leader_election.cc:288] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df35365ecb2d431090fb98cade9df0a8 (127.5.198.193:42527), 5bdb606e9ba2492f94c9f32eacd6f293 (127.5.198.195:38781)
I20260929 01:07:10.636272 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.629868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:45825 (local address 127.5.198.195:38781)
0929 01:07:10.630074 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:10.630083 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:10.631233 (+ 1150us) server_negotiation.cc:407] Connection header received
0929 01:07:10.631392 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:10.631397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:10.631462 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:10.631551 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:10.632133 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.632911 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.634615 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.635349 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.635519 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.635603 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:10.635893 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 01:07:10.636018 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 01:07:10.636274 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.629766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:59353 (local address 127.5.198.193:42527)
0929 01:07:10.629987 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:10.629992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:10.631122 (+ 1130us) server_negotiation.cc:407] Connection header received
0929 01:07:10.631338 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:10.631347 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:10.631403 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:10.631495 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:10.632025 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.632783 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.634622 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.635361 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.635549 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.635627 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:10.635904 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 01:07:10.636035 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 01:07:10.636451 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.629885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:38781 (local address 127.5.198.194:45825)
0929 01:07:10.631141 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:10.631155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:10.631258 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:10.631562 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:10.631566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:10.631592 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:10.631993 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:10.632001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.633055 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.633068 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:10.634454 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:10.634484 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.635583 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.635590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:10.635738 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.635764 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 01:07:10.635857 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 01:07:10.635987 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"mutex_wait_us":4,"thread_start_us":396,"threads_started":1}
I20260929 01:07:10.636502 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.629545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:42527 (local address 127.5.198.194:59353)
0929 01:07:10.631017 (+ 1472us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:10.631036 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:10.631212 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:10.631509 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:10.631513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:10.631536 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:10.631887 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:10.631895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.632901 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.632909 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:10.634457 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:10.634493 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.635617 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.635623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:10.635762 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.635783 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:07:10.635881 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 01:07:10.636011 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1333,"mutex_wait_us":76,"thread_start_us":1366,"threads_started":1}
I20260929 01:07:10.637563 6901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c940bac338a54f6fa1e14b9836bc6698" candidate_uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bdb606e9ba2492f94c9f32eacd6f293" is_pre_election: true
I20260929 01:07:10.637563 6634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c940bac338a54f6fa1e14b9836bc6698" candidate_uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df35365ecb2d431090fb98cade9df0a8" is_pre_election: true
I20260929 01:07:10.638126 6634 raft_consensus.cc:2454] T c940bac338a54f6fa1e14b9836bc6698 P df35365ecb2d431090fb98cade9df0a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92bb223cbcbe4d5f8b9543d8b1995b2c in term 0.
I20260929 01:07:10.638124 6901 raft_consensus.cc:2454] T c940bac338a54f6fa1e14b9836bc6698 P 5bdb606e9ba2492f94c9f32eacd6f293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92bb223cbcbe4d5f8b9543d8b1995b2c in term 0.
I20260929 01:07:10.639197 6702 leader_election.cc:302] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bdb606e9ba2492f94c9f32eacd6f293, 92bb223cbcbe4d5f8b9543d8b1995b2c; no voters:
I20260929 01:07:10.639644 6970 raft_consensus.cc:2781] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:07:10.639787 6970 raft_consensus.cc:482] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:07:10.639895 6970 raft_consensus.cc:3037] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:10.642059 6970 raft_consensus.cc:504] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df35365ecb2d431090fb98cade9df0a8" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 42527 } } peers { permanent_uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46491 } } peers { permanent_uuid: "5bdb606e9ba2492f94c9f32eacd6f293" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38781 } }
I20260929 01:07:10.642694 6970 leader_election.cc:288] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [CANDIDATE]: Term 1 election: Requested vote from peers df35365ecb2d431090fb98cade9df0a8 (127.5.198.193:42527), 5bdb606e9ba2492f94c9f32eacd6f293 (127.5.198.195:38781)
I20260929 01:07:10.643144 6634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c940bac338a54f6fa1e14b9836bc6698" candidate_uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df35365ecb2d431090fb98cade9df0a8"
I20260929 01:07:10.643186 6901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c940bac338a54f6fa1e14b9836bc6698" candidate_uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bdb606e9ba2492f94c9f32eacd6f293"
I20260929 01:07:10.643337 6634 raft_consensus.cc:3037] T c940bac338a54f6fa1e14b9836bc6698 P df35365ecb2d431090fb98cade9df0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:10.643466 6901 raft_consensus.cc:3037] T c940bac338a54f6fa1e14b9836bc6698 P 5bdb606e9ba2492f94c9f32eacd6f293 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:10.645597 6634 raft_consensus.cc:2454] T c940bac338a54f6fa1e14b9836bc6698 P df35365ecb2d431090fb98cade9df0a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92bb223cbcbe4d5f8b9543d8b1995b2c in term 1.
I20260929 01:07:10.645798 6901 raft_consensus.cc:2454] T c940bac338a54f6fa1e14b9836bc6698 P 5bdb606e9ba2492f94c9f32eacd6f293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92bb223cbcbe4d5f8b9543d8b1995b2c in term 1.
I20260929 01:07:10.646055 6701 leader_election.cc:302] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92bb223cbcbe4d5f8b9543d8b1995b2c, df35365ecb2d431090fb98cade9df0a8; no voters:
I20260929 01:07:10.646338 6970 raft_consensus.cc:2781] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:10.646781 6970 raft_consensus.cc:686] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [term 1 LEADER]: Becoming Leader. State: Replica: 92bb223cbcbe4d5f8b9543d8b1995b2c, State: Running, Role: LEADER
I20260929 01:07:10.647228 6970 consensus_queue.cc:237] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df35365ecb2d431090fb98cade9df0a8" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 42527 } } peers { permanent_uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46491 } } peers { permanent_uuid: "5bdb606e9ba2492f94c9f32eacd6f293" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38781 } }
I20260929 01:07:10.651947 6489 catalog_manager.cc:5495] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 92bb223cbcbe4d5f8b9543d8b1995b2c (127.5.198.194). New cstate: current_term: 1 leader_uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df35365ecb2d431090fb98cade9df0a8" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92bb223cbcbe4d5f8b9543d8b1995b2c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bdb606e9ba2492f94c9f32eacd6f293" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38781 } health_report { overall_health: UNKNOWN } } }
Sep 29 01:07:10 dist-test-slave-41cl krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
W20260929 01:07:10.693156 6947 tablet.cc:2369] T c940bac338a54f6fa1e14b9836bc6698 P 5bdb606e9ba2492f94c9f32eacd6f293: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:07:10.699719 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.684599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35258 (local address 127.5.198.193:42527)
0929 01:07:10.684774 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:10.684778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:10.685294 (+ 516us) server_negotiation.cc:407] Connection header received
0929 01:07:10.685431 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:10.685436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:10.685518 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:10.685609 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:10.687002 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.687745 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.689390 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.689864 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.689958 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.694025 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:10.694045 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:10.694062 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:10.694100 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:10.696998 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:10.697658 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:10.697664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:10.697666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:10.697782 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:10.698202 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:10.698207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:10.698210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:10.698697 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:10.698983 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:10.699229 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 01:07:10.699443 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 01:07:10.718149 6814 tablet.cc:2369] T c940bac338a54f6fa1e14b9836bc6698 P 92bb223cbcbe4d5f8b9543d8b1995b2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 01:07:10 dist-test-slave-41cl krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644030, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260929 01:07:10.728736 6680 tablet.cc:2369] T c940bac338a54f6fa1e14b9836bc6698 P df35365ecb2d431090fb98cade9df0a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 01:07:10 dist-test-slave-41cl krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644030, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:07:10.742995 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.730418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.5.198.193:42527)
0929 01:07:10.730620 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:10.730629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:10.730969 (+ 340us) server_negotiation.cc:407] Connection header received
0929 01:07:10.731118 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:10.731122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:10.731184 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:10.731267 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:10.732537 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.733300 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.734501 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.734925 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.735018 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.738535 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:10.738540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:10.738543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:10.738572 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:10.741024 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:10.741535 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:10.741538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:10.741541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:10.741645 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:10.741995 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:10.741998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:10.742000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:10.742219 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:10.742411 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:10.742596 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 01:07:10.742738 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 01:07:10 dist-test-slave-41cl krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 01:07:10 dist-test-slave-41cl krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:07:10.785948 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.772061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.5.198.193:42527)
0929 01:07:10.772275 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:10.772282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:10.772531 (+ 249us) server_negotiation.cc:407] Connection header received
0929 01:07:10.772663 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:10.772667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:10.772728 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:10.772816 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:10.774552 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.775246 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.776424 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.776931 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.777027 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.780663 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:10.780669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:10.780672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:10.780711 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:10.783545 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:10.784152 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:10.784158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:10.784160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:10.784284 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:10.784683 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:10.784687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:10.784689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:10.785016 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:10.785204 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:10.785451 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 01:07:10.785619 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 01:07:10.787889 6594 tablet_service.cc:2908] Rejecting scan request for tablet c940bac338a54f6fa1e14b9836bc6698: Uninitialized: clean time has not yet been initialized
I20260929 01:07:10.790700 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 6551
I20260929 01:07:10.803412 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.791462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.5.198.193:42527)
0929 01:07:10.791639 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:10.791643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:10.792047 (+ 404us) server_negotiation.cc:407] Connection header received
0929 01:07:10.792137 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:10.792141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:10.792205 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:10.792283 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:10.793367 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.794007 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.795135 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.795593 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.795704 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.797938 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:10.797943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:10.797983 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:10.798017 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:10.801187 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:10.801753 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:10.801757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:10.801759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:10.801911 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:10.802316 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:10.802321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:10.802325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:10.802601 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:10.802873 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:10.803041 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 01:07:10.803196 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:07:10.804031 6674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:10.892063 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 6551
I20260929 01:07:10.909164 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 6685
Sep 29 01:07:10 dist-test-slave-41cl krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:07:10.924268 6962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:10.910057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.5.198.194:46491)
0929 01:07:10.910264 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:10.910271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:10.910291 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:07:10.910354 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:10.910366 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:10.910441 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:10.910540 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:10.911673 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.912925 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.914160 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:10.914769 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:10.914878 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:10.918900 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:10.918915 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:10.918930 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:10.918984 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:10.921921 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:10.922575 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:10.922582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:10.922585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:10.922708 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:10.923065 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:10.923068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:10.923070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:10.923386 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:10.923600 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:10.923837 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 01:07:10.924031 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:07:10.924983 6808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:11.013418 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 6685
I20260929 01:07:11.030055 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 6818
Sep 29 01:07:11 dist-test-slave-41cl krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:07:11.044373 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:11.030841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55898 (local address 127.5.198.195:38781)
0929 01:07:11.031082 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:11.031089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:11.031109 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:07:11.031238 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:11.031242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:11.031302 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:11.031411 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:11.032588 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:11.033302 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:11.034550 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:11.034930 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:11.035059 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:11.039216 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:11.039229 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:11.039245 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:11.039275 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:11.041899 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:11.042536 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:11.042544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:11.042547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:11.042680 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:11.043036 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:11.043043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:11.043046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:11.043373 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:11.043660 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:11.043962 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 01:07:11.044137 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260929 01:07:11.045152 6941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:11.133397 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 6818
I20260929 01:07:11.151244 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 6458
I20260929 01:07:11.152338 6520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:11.260728 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 6458
2026-09-29T01:07:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4043 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:07:11 dist-test-slave-41cl krb5kdc[6994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:07:11 dist-test-slave-41cl krb5kdc[6994](info): set up 2 sockets
Sep 29 01:07:11 dist-test-slave-41cl krb5kdc[6994](info): commencing operation
krb5kdc: starting...
W20260929 01:07:12.404830 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:07:12 dist-test-slave-41cl krb5kdc[6994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:07:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:07:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:07:12.592738 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:44003
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:44003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:12.903764 7007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:12.904098 7007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:12.904263 7007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:12.911454 7007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:12.911587 7007 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 01:07:12.911679 7007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:12.911800 7007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:12.911893 7007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:12.911978 7007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:07:12.921978 7007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:44003
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:44003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:12.923337 7007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:12.924655 7007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:12.936033 7012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:12.936481 7013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:12.936088 7015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:12.936945 7007 server_base.cc:1034] running on GCE node
I20260929 01:07:12.937708 7007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:12.938999 7007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:12.940258 7007 hybrid_clock.cc:648] HybridClock initialized: now 1790644032940191 us; error 40 us; skew 500 ppm
I20260929 01:07:12.940670 7007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:12 dist-test-slave-41cl krb5kdc[6994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644032, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:12.943658 7007 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:07:12.945150 7007 webserver.cc:460] Webserver started at http://127.5.198.254:44315/ using document root <none> and password file <none>
I20260929 01:07:12.945776 7007 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:12.945869 7007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:12.946117 7007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:12.949098 7007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "99709fd2347d4185b4ef5b3a597fd2c7"
format_stamp: "Formatted at 2026-09-29 01:07:12 on dist-test-slave-41cl"
server_key: "cd05baaed017f5f7c002c42b7bb92e5a"
server_key_iv: "52c467343ef4f442cf8df703e8e39729"
server_key_version: "encryptionkey@0"
I20260929 01:07:12.949801 7007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "99709fd2347d4185b4ef5b3a597fd2c7"
format_stamp: "Formatted at 2026-09-29 01:07:12 on dist-test-slave-41cl"
server_key: "cd05baaed017f5f7c002c42b7bb92e5a"
server_key_iv: "52c467343ef4f442cf8df703e8e39729"
server_key_version: "encryptionkey@0"
I20260929 01:07:12.954598 7007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:07:12.957901 7022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:12.959268 7007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:07:12.959456 7007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "99709fd2347d4185b4ef5b3a597fd2c7"
format_stamp: "Formatted at 2026-09-29 01:07:12 on dist-test-slave-41cl"
server_key: "cd05baaed017f5f7c002c42b7bb92e5a"
server_key_iv: "52c467343ef4f442cf8df703e8e39729"
server_key_version: "encryptionkey@0"
I20260929 01:07:12.959669 7007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:12.981643 7007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:12.991731 7007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:12.992015 7007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:13.013285 7007 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:44003
I20260929 01:07:13.013303 7083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:44003 every 8 connection(s)
I20260929 01:07:13.014640 7007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:07:13.018332 7084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:13.018988 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 7007
I20260929 01:07:13.019215 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:07:13.020028 5915 external_mini_cluster.cc:1428] Setting key e72f9084fa3ddfddea28ee0151930470
I20260929 01:07:13.032109 7084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7: Bootstrap starting.
Sep 29 01:07:13 dist-test-slave-41cl krb5kdc[6994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:13.035032 7084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:13.036314 7084 log.cc:824] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:13.039754 7084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7: No bootstrap required, opened a new log
I20260929 01:07:13.043949 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:13.022020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.5.198.254:44003)
0929 01:07:13.022801 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:13.022817 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:13.022868 (+ 51us) server_negotiation.cc:407] Connection header received
0929 01:07:13.024828 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:13.024855 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:13.025366 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:13.025810 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:13.027106 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:13.029135 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:13.030407 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:13.031403 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:13.031573 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:13.035465 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:13.035478 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:13.035496 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:13.035539 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:13.039594 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:13.040274 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:13.040282 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:13.040291 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:13.040530 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:13.040864 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:13.040868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:13.040871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:13.041535 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:13.041824 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:13.042208 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 01:07:13.042625 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":305,"threads_started":1}
I20260929 01:07:13.049371 7084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99709fd2347d4185b4ef5b3a597fd2c7" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44003 } }
I20260929 01:07:13.049682 7084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:13.049754 7084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99709fd2347d4185b4ef5b3a597fd2c7, State: Initialized, Role: FOLLOWER
I20260929 01:07:13.050215 7084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99709fd2347d4185b4ef5b3a597fd2c7" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44003 } }
I20260929 01:07:13.050401 7084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:07:13.050494 7084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:07:13.050618 7084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:13.052678 7084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99709fd2347d4185b4ef5b3a597fd2c7" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44003 } }
I20260929 01:07:13.053285 7084 leader_election.cc:302] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99709fd2347d4185b4ef5b3a597fd2c7; no voters:
I20260929 01:07:13.053844 7084 leader_election.cc:288] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:07:13.054008 7089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:13.054706 7089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [term 1 LEADER]: Becoming Leader. State: Replica: 99709fd2347d4185b4ef5b3a597fd2c7, State: Running, Role: LEADER
I20260929 01:07:13.055250 7089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99709fd2347d4185b4ef5b3a597fd2c7" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44003 } }
I20260929 01:07:13.055918 7084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:13.058274 7091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99709fd2347d4185b4ef5b3a597fd2c7. Latest consensus state: current_term: 1 leader_uuid: "99709fd2347d4185b4ef5b3a597fd2c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99709fd2347d4185b4ef5b3a597fd2c7" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44003 } } }
I20260929 01:07:13.058694 7091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:13.059798 7090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99709fd2347d4185b4ef5b3a597fd2c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99709fd2347d4185b4ef5b3a597fd2c7" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44003 } } }
I20260929 01:07:13.060099 7090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:13.060904 7096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:07:13.065708 7096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:07:13.074120 7096 catalog_manager.cc:1347] Generated new cluster ID: 55f60098458f4a99899c60a11ddcc8ba
I20260929 01:07:13.074398 7096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:07:13.100945 7096 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:07:13.102849 7096 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:07:13.110458 7096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7: Generated new TSK 0
I20260929 01:07:13.111464 7096 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:07:13.164556 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44003
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:13.493746 7110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:13.494004 7110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:13.494156 7110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:13.501324 7110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:13.501422 7110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:13.501570 7110 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:07:13.501694 7110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:07:13.510887 7110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.198.254:44003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:13.512102 7110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:13.513392 7110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:13.525906 7116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:13.526098 7118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:13.526543 7110 server_base.cc:1034] running on GCE node
W20260929 01:07:13.526682 7115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:13.527290 7110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:13.528398 7110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:13.529683 7110 hybrid_clock.cc:648] HybridClock initialized: now 1790644033529609 us; error 51 us; skew 500 ppm
I20260929 01:07:13.530016 7110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:13 dist-test-slave-41cl krb5kdc[6994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644033, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:13.533213 7110 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:07:13.534787 7110 webserver.cc:460] Webserver started at http://127.5.198.193:32897/ using document root <none> and password file <none>
I20260929 01:07:13.535437 7110 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:13.535523 7110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:13.535863 7110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:13.538764 7110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "a05ea8a14fa845dfbb729b0d89ef7612"
format_stamp: "Formatted at 2026-09-29 01:07:13 on dist-test-slave-41cl"
server_key: "2c23567bc74f0da80e0213baf3835dd7"
server_key_iv: "1eb6ac2af80baf2275f258474099c698"
server_key_version: "encryptionkey@0"
I20260929 01:07:13.539515 7110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "a05ea8a14fa845dfbb729b0d89ef7612"
format_stamp: "Formatted at 2026-09-29 01:07:13 on dist-test-slave-41cl"
server_key: "2c23567bc74f0da80e0213baf3835dd7"
server_key_iv: "1eb6ac2af80baf2275f258474099c698"
server_key_version: "encryptionkey@0"
I20260929 01:07:13.544278 7110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260929 01:07:13.547684 7125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:13.549036 7110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:07:13.549224 7110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "a05ea8a14fa845dfbb729b0d89ef7612"
format_stamp: "Formatted at 2026-09-29 01:07:13 on dist-test-slave-41cl"
server_key: "2c23567bc74f0da80e0213baf3835dd7"
server_key_iv: "1eb6ac2af80baf2275f258474099c698"
server_key_version: "encryptionkey@0"
I20260929 01:07:13.549437 7110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:13.559815 7110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:13.563722 7110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:13.563974 7110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:13.566459 7110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:13.566589 7110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:13.566709 7110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:13.566776 7110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:07:13 dist-test-slave-41cl krb5kdc[6994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644033, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:13.592711 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:13.567792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44003 (local address 127.5.198.193:51825)
0929 01:07:13.568705 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:13.568742 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:13.570615 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:13.571362 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:13.571378 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:13.572196 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:13.573015 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:13.573036 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:13.574484 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:13.574493 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:13.576844 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:13.576856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:13.577675 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:13.577684 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:13.578047 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:13.578923 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:13.578947 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:13.581689 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:13.586655 (+ 4966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:13.586668 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:13.586694 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:13.587641 (+ 947us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:13.588124 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:13.588133 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:13.588140 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:13.588514 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:13.589527 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:13.589534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:13.589969 (+ 435us) client_negotiation.cc:769] Sending connection context
0929 01:07:13.590375 (+ 406us) client_negotiation.cc:240] Negotiation successful
0929 01:07:13.591003 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":377,"threads_started":1}
I20260929 01:07:13.593287 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:13.567176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:51825 (local address 127.5.198.254:44003)
0929 01:07:13.567636 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:13.567641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:13.568843 (+ 1202us) server_negotiation.cc:407] Connection header received
0929 01:07:13.570866 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:13.570871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:13.570960 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:13.571081 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:13.573253 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:13.574328 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:13.577030 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:13.577443 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:13.577587 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:13.581985 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:13.581992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:13.581997 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:13.582041 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:13.586457 (+ 4416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:13.587816 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:13.587823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:13.587829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:13.587987 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:13.588790 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:13.588799 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:13.588805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:13.589326 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:13.589564 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:13.592683 (+ 3119us) server_negotiation.cc:299] Negotiation successful
0929 01:07:13.592941 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":262,"threads_started":1}
I20260929 01:07:13.617446 7110 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:34655
I20260929 01:07:13.617528 7245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:34655 every 8 connection(s)
I20260929 01:07:13.619027 7110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:07:13.620426 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:13.600808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:37143 (local address 127.5.198.254:44003)
0929 01:07:13.601012 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:13.601019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:13.605288 (+ 4269us) server_negotiation.cc:407] Connection header received
0929 01:07:13.605490 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:13.605498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:13.605589 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:13.605705 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:13.607393 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:13.608252 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:13.610438 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:13.611155 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:13.611304 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:13.614233 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:13.614246 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:13.614250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:13.614277 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:13.617693 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:13.618433 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:13.618437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:13.618439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:13.618571 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:13.618942 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:13.618950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:13.618952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:13.619193 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:13.619297 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:13.619853 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 01:07:13.620112 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 01:07:13.621330 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:13.600607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44003 (local address 127.5.198.193:37143)
0929 01:07:13.605161 (+ 4554us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:13.605185 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:13.605347 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:13.605916 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:13.605925 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:13.606697 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:13.607249 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:13.607262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:13.608420 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:13.608424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:13.610305 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:13.610317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:13.611437 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:13.611451 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:13.611689 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:13.612539 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:13.612566 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:13.614030 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:13.617810 (+ 3780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:13.617815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:13.617819 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:13.618308 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:13.618683 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:13.618688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:13.618691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:13.618839 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:13.619302 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:13.619307 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:13.619512 (+ 205us) client_negotiation.cc:769] Sending connection context
0929 01:07:13.619736 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 01:07:13.620215 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4377,"thread_start_us":4155,"threads_started":1}
I20260929 01:07:13.621371 7246 heartbeater.cc:347] Connected to a master server at 127.5.198.254:44003
I20260929 01:07:13.621632 7246 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:13.622442 7246 heartbeater.cc:511] Master 127.5.198.254:44003 requested a full tablet report, sending...
I20260929 01:07:13.624691 7039 ts_manager.cc:194] Registered new tserver with Master: a05ea8a14fa845dfbb729b0d89ef7612 (127.5.198.193:34655)
I20260929 01:07:13.626068 7039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:51825
I20260929 01:07:13.627617 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 7110
I20260929 01:07:13.627840 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:07:13.628443 5915 external_mini_cluster.cc:1428] Setting key 06097c51ed65278224283990d9a977fd
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:07:13.667503 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44003
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:13.980947 7250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:13.981204 7250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:13.981297 7250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:13.988211 7250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:13.988312 7250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:13.988395 7250 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:07:13.988490 7250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:07:13.998070 7250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.198.254:44003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:13.999221 7250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:14.000524 7250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:14.013340 7255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:14.013522 7258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:14.016644 7250 server_base.cc:1034] running on GCE node
W20260929 01:07:14.016757 7256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:14.017428 7250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:14.018385 7250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:14.019634 7250 hybrid_clock.cc:648] HybridClock initialized: now 1790644034019566 us; error 45 us; skew 500 ppm
I20260929 01:07:14.019969 7250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:14 dist-test-slave-41cl krb5kdc[6994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644034, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:14.028419 7250 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:07:14.029824 7250 webserver.cc:460] Webserver started at http://127.5.198.194:34895/ using document root <none> and password file <none>
I20260929 01:07:14.030499 7250 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:14.030633 7250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:14.030946 7250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:14.033826 7250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6"
format_stamp: "Formatted at 2026-09-29 01:07:14 on dist-test-slave-41cl"
server_key: "ff26f426d05cd4ab753dca0bcb8f42bd"
server_key_iv: "58cd39f4bf7da2ded59a2ddeea84038d"
server_key_version: "encryptionkey@0"
I20260929 01:07:14.034570 7250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6"
format_stamp: "Formatted at 2026-09-29 01:07:14 on dist-test-slave-41cl"
server_key: "ff26f426d05cd4ab753dca0bcb8f42bd"
server_key_iv: "58cd39f4bf7da2ded59a2ddeea84038d"
server_key_version: "encryptionkey@0"
I20260929 01:07:14.039597 7250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:07:14.043057 7265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:14.044313 7250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:07:14.044519 7250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6"
format_stamp: "Formatted at 2026-09-29 01:07:14 on dist-test-slave-41cl"
server_key: "ff26f426d05cd4ab753dca0bcb8f42bd"
server_key_iv: "58cd39f4bf7da2ded59a2ddeea84038d"
server_key_version: "encryptionkey@0"
I20260929 01:07:14.044726 7250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:14.059906 7250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:14.063710 7250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:14.063997 7250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:14.066543 7250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:14.066650 7250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:14.066793 7250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:14.066874 7250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:07:14 dist-test-slave-41cl krb5kdc[6994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644034, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:14.088848 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.067249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:59107 (local address 127.5.198.254:44003)
0929 01:07:14.067423 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:14.067429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:14.068987 (+ 1558us) server_negotiation.cc:407] Connection header received
0929 01:07:14.072129 (+ 3142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:14.072138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:14.072242 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:14.072381 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:14.074177 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.074889 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.077520 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.077962 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.078093 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.081633 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:14.081638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:14.081641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:14.081670 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:14.085544 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:14.086522 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:14.086530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:14.086535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:14.086699 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:14.087188 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:14.087193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:14.087197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:14.087469 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:14.087631 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:14.088266 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 01:07:14.088505 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:07:14.089805 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.067884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44003 (local address 127.5.198.194:59107)
0929 01:07:14.068809 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:14.068864 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:14.071650 (+ 2786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:14.072674 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:14.072685 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:14.073340 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:14.073930 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.073948 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.075033 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.075039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.077375 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.077395 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.078273 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.078281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.078463 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.079110 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:14.079130 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:14.081413 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:14.085712 (+ 4299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:14.085728 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:14.085745 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:14.086376 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:14.086827 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:14.086833 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:14.086837 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:14.087056 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:14.087598 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:14.087608 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:14.087935 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 01:07:14.088179 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 01:07:14.088707 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":366,"threads_started":1}
I20260929 01:07:14.109568 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.096006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:33931 (local address 127.5.198.254:44003)
0929 01:07:14.096194 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:14.096199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:14.096901 (+ 702us) server_negotiation.cc:407] Connection header received
0929 01:07:14.097121 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:14.097126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:14.097196 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:14.097293 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:14.098137 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.098826 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.100090 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.100573 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.100720 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.103036 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:14.103045 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:14.103052 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:14.103095 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:14.106636 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:14.107546 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:14.107556 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:14.107558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:14.107673 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:14.108037 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:14.108045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:14.108049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:14.108441 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:14.108616 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:14.108997 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 01:07:14.109209 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 01:07:14.109668 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.096091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44003 (local address 127.5.198.194:33931)
0929 01:07:14.096802 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:14.096827 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:14.096969 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:14.097311 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:14.097315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:14.097738 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:14.098003 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.098011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.098973 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.098978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.099952 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.099965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.100768 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.100774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.100970 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.101649 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:14.101677 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:14.102762 (+ 1085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:14.106809 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:14.106816 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:14.106821 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:14.107379 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:14.107792 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:14.107797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:14.107800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:14.107922 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:14.108584 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:14.108589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:14.108766 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 01:07:14.108905 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 01:07:14.109208 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":346,"threads_started":1}
I20260929 01:07:14.123966 7250 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:45467
I20260929 01:07:14.124004 7384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:45467 every 8 connection(s)
I20260929 01:07:14.125303 7250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:07:14.127255 7385 heartbeater.cc:347] Connected to a master server at 127.5.198.254:44003
I20260929 01:07:14.127547 7385 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:14.128229 7385 heartbeater.cc:511] Master 127.5.198.254:44003 requested a full tablet report, sending...
I20260929 01:07:14.129516 7039 ts_manager.cc:194] Registered new tserver with Master: fbc3ad18ebdb4d168516d49c00c7f0e6 (127.5.198.194:45467)
I20260929 01:07:14.130316 7039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:59107
I20260929 01:07:14.133096 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 7250
I20260929 01:07:14.133282 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:07:14.133692 5915 external_mini_cluster.cc:1428] Setting key d50cde0cfa76fe815f17e021e1a56897
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:07:14.172071 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44003
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:14.483649 7389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:14.483907 7389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:14.483999 7389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:14.490954 7389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:14.491070 7389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:14.491151 7389 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:07:14.491231 7389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:07:14.500946 7389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.198.254:44003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:14.502100 7389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:14.503301 7389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:14.515992 7395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:14.516032 7397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:14.516054 7394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:14.517076 7389 server_base.cc:1034] running on GCE node
I20260929 01:07:14.517657 7389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:14.518532 7389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:14.519774 7389 hybrid_clock.cc:648] HybridClock initialized: now 1790644034519728 us; error 42 us; skew 500 ppm
I20260929 01:07:14.520066 7389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:14 dist-test-slave-41cl krb5kdc[6994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644034, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:14.523319 7389 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:07:14.524709 7389 webserver.cc:460] Webserver started at http://127.5.198.195:36317/ using document root <none> and password file <none>
I20260929 01:07:14.525409 7389 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:14.525535 7389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:14.525843 7389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:14.528759 7389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e"
format_stamp: "Formatted at 2026-09-29 01:07:14 on dist-test-slave-41cl"
server_key: "28c6af16f11ec058a17c9536c76a576a"
server_key_iv: "dec527fec196d9a2c280f97e1bfd5ddf"
server_key_version: "encryptionkey@0"
I20260929 01:07:14.529532 7389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e"
format_stamp: "Formatted at 2026-09-29 01:07:14 on dist-test-slave-41cl"
server_key: "28c6af16f11ec058a17c9536c76a576a"
server_key_iv: "dec527fec196d9a2c280f97e1bfd5ddf"
server_key_version: "encryptionkey@0"
I20260929 01:07:14.534543 7389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:07:14.537879 7404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:14.539258 7389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:07:14.539440 7389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e"
format_stamp: "Formatted at 2026-09-29 01:07:14 on dist-test-slave-41cl"
server_key: "28c6af16f11ec058a17c9536c76a576a"
server_key_iv: "dec527fec196d9a2c280f97e1bfd5ddf"
server_key_version: "encryptionkey@0"
I20260929 01:07:14.539665 7389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:14.577458 7389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:14.582089 7389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:14.582829 7389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:14.587064 7389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:14.587206 7389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:14.587391 7389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:14.587517 7389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 29 01:07:14 dist-test-slave-41cl krb5kdc[6994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644034, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:14.616966 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.588867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:50569 (local address 127.5.198.254:44003)
0929 01:07:14.589174 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:14.589181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:14.591364 (+ 2183us) server_negotiation.cc:407] Connection header received
0929 01:07:14.594288 (+ 2924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:14.594293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:14.594389 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:14.594533 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:14.597778 (+ 3245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.599099 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.602808 (+ 3709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.603588 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.603709 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.609271 (+ 5562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:14.609276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:14.609279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:14.609314 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:14.612586 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:14.613602 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:14.613609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:14.613614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:14.613774 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:14.614383 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:14.614392 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:14.614398 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:14.614929 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:14.615120 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:14.616338 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0929 01:07:14.616628 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260929 01:07:14.618937 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.589744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44003 (local address 127.5.198.195:50569)
0929 01:07:14.591137 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:14.591215 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:14.593829 (+ 2614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:14.594943 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:14.594955 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:14.596113 (+ 1158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:14.597373 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.597415 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.599376 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.599385 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.602600 (+ 3215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.602617 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.604018 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.604031 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.604272 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.605134 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:14.605201 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:14.608984 (+ 3783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:14.612718 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:14.612728 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:14.612749 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:14.613382 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:14.613898 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:14.613903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:14.613916 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:14.614212 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:14.615212 (+ 1000us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:14.615225 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:14.615833 (+ 608us) client_negotiation.cc:769] Sending connection context
0929 01:07:14.616214 (+ 381us) client_negotiation.cc:240] Negotiation successful
0929 01:07:14.616978 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"thread_start_us":602,"threads_started":1}
I20260929 01:07:14.628708 7246 heartbeater.cc:503] Master 127.5.198.254:44003 was elected leader, sending a full tablet report...
I20260929 01:07:14.633610 7389 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:39053
I20260929 01:07:14.633641 7523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:39053 every 8 connection(s)
I20260929 01:07:14.635013 7389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:07:14.637506 7524 heartbeater.cc:347] Connected to a master server at 127.5.198.254:44003
I20260929 01:07:14.637753 7524 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:14.638571 7524 heartbeater.cc:511] Master 127.5.198.254:44003 requested a full tablet report, sending...
I20260929 01:07:14.638998 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 7389
I20260929 01:07:14.639199 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:07:14.639772 5915 external_mini_cluster.cc:1428] Setting key 02ec853cdb34ea728b56bf1ced407d40
I20260929 01:07:14.640137 7039 ts_manager.cc:194] Registered new tserver with Master: 1c2cfeb4589b4e67b66d147c66bb0c0e (127.5.198.195:39053)
I20260929 01:07:14.641297 7039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:50569
I20260929 01:07:14.643162 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:07:14.643700 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.626964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:51167 (local address 127.5.198.254:44003)
0929 01:07:14.627168 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:14.627174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:14.627835 (+ 661us) server_negotiation.cc:407] Connection header received
0929 01:07:14.628172 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:14.628179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:14.628267 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:14.628371 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:14.629506 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.630593 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.632368 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.633157 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.633307 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.635185 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:14.635193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:14.635197 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:14.635232 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:14.640777 (+ 5545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:14.641456 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:14.641463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:14.641467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:14.641660 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:14.642083 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:14.642092 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:14.642095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:14.642442 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:14.642644 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:14.643102 (+ 458us) server_negotiation.cc:299] Negotiation successful
0929 01:07:14.643327 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:07:14.643891 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.626943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44003 (local address 127.5.198.195:51167)
0929 01:07:14.627710 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:14.627736 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:14.627964 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:14.628394 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:14.628398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:14.628917 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:14.629337 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.629352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.630751 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.630758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.632219 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.632230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.633360 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.633370 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.633603 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.634256 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:14.634272 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:14.634990 (+ 718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:14.640913 (+ 5923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:14.640918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:14.640922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:14.641333 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:14.641781 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:14.641785 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:14.641787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:14.641950 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:14.642603 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:14.642608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:14.642845 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 01:07:14.642997 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 01:07:14.643406 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":90,"thread_start_us":355,"threads_started":1}
Sep 29 01:07:14 dist-test-slave-41cl krb5kdc[6994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644034, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 01:07:14 dist-test-slave-41cl krb5kdc[6994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644034, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:14.688922 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.675182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.5.198.254:44003)
0929 01:07:14.675423 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:14.675432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:14.675665 (+ 233us) server_negotiation.cc:407] Connection header received
0929 01:07:14.675909 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:14.675913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:14.675978 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:14.676087 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:14.677537 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.678293 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.679532 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.679959 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.680086 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.683890 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:14.683895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:14.683898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:14.683930 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:14.686421 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:14.687017 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:14.687024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:14.687028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:14.687146 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:14.687525 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:14.687531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:14.687534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:14.687793 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:14.688042 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:14.688317 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 01:07:14.688552 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 01:07:14.693104 7039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:07:14.699739 7039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:07:14.726085 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.711770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.5.198.194:45467)
0929 01:07:14.712304 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:14.712316 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:14.714191 (+ 1875us) server_negotiation.cc:407] Connection header received
0929 01:07:14.714676 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:14.714689 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:14.715037 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:14.715338 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:14.716595 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.718566 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.722173 (+ 3607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.724097 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.724240 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.724547 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:14.725444 (+ 897us) server_negotiation.cc:299] Negotiation successful
0929 01:07:14.725683 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":699,"threads_started":1}
I20260929 01:07:14.727443 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.711648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:45467 (local address 127.0.0.1:48566)
0929 01:07:14.714003 (+ 2355us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:14.714053 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:14.714398 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:14.715739 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:14.715751 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:14.715791 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:14.716295 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.716312 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.718825 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.718835 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.721957 (+ 3122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.721977 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.724775 (+ 2798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.724784 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.725087 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.725160 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 01:07:14.726722 (+ 1562us) client_negotiation.cc:240] Negotiation successful
0929 01:07:14.726997 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2046,"thread_start_us":1870,"threads_started":1}
I20260929 01:07:14.731479 7315 tablet_service.cc:1467] Processing CreateTablet for tablet af0b1d88e49c422db28a08e9a8ee8775 (DEFAULT_TABLE table=test-table [id=7808d90b84ac4a46986fa67b7b81400b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:14.732955 7315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0b1d88e49c422db28a08e9a8ee8775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:14.734715 7540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.712249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34846 (local address 127.5.198.195:39053)
0929 01:07:14.727650 (+ 15401us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:14.727658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:14.727691 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:07:14.727784 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:14.727794 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:14.728041 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:14.728246 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:14.728939 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.730436 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.732259 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.733596 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.733725 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.733960 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:14.734199 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 01:07:14.734363 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15240,"thread_start_us":15088,"threads_started":1}
I20260929 01:07:14.735348 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.711708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:39053 (local address 127.0.0.1:34846)
0929 01:07:14.725710 (+ 14002us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:14.725733 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:14.725868 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:14.728231 (+ 2363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:14.728239 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:14.728265 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:14.728714 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.728728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.730570 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.730576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.732042 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.732052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.733806 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.733813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.734007 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.734044 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 01:07:14.734140 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 01:07:14.734268 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13825,"mutex_wait_us":20,"thread_start_us":14256,"threads_started":1}
I20260929 01:07:14.735947 7542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.715346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:34655 (local address 127.0.0.1:52578)
0929 01:07:14.726584 (+ 11238us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:14.726606 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:14.726736 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:14.728004 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:14.728009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:14.728032 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:14.728388 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.728404 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.730351 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.730357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.732556 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.732570 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.735032 (+ 2462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.735038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.735192 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.735224 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 01:07:14.735307 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 01:07:14.735444 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11087,"mutex_wait_us":27,"thread_start_us":10944,"threads_started":1}
I20260929 01:07:14.736133 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.715001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52578 (local address 127.5.198.193:34655)
0929 01:07:14.727229 (+ 12228us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:14.727239 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:14.727272 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:07:14.727375 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:14.727384 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:14.727669 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:14.727865 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:14.728620 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.730207 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.732724 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.734548 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.734683 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.734889 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:14.735609 (+ 720us) server_negotiation.cc:299] Negotiation successful
0929 01:07:14.735795 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12070,"thread_start_us":11935,"threads_started":1}
I20260929 01:07:14.738948 7454 tablet_service.cc:1467] Processing CreateTablet for tablet af0b1d88e49c422db28a08e9a8ee8775 (DEFAULT_TABLE table=test-table [id=7808d90b84ac4a46986fa67b7b81400b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:14.740085 7454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0b1d88e49c422db28a08e9a8ee8775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:14.739557 7176 tablet_service.cc:1467] Processing CreateTablet for tablet af0b1d88e49c422db28a08e9a8ee8775 (DEFAULT_TABLE table=test-table [id=7808d90b84ac4a46986fa67b7b81400b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:14.740885 7176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0b1d88e49c422db28a08e9a8ee8775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:14.747396 7543 tablet_bootstrap.cc:492] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6: Bootstrap starting.
I20260929 01:07:14.751127 7543 tablet_bootstrap.cc:654] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:14.752514 7544 tablet_bootstrap.cc:492] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Bootstrap starting.
I20260929 01:07:14.752578 7543 log.cc:824] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:14.755985 7544 tablet_bootstrap.cc:654] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:14.756140 7543 tablet_bootstrap.cc:492] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6: No bootstrap required, opened a new log
I20260929 01:07:14.756381 7543 ts_tablet_manager.cc:1397] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 01:07:14.757093 7544 log.cc:824] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:14.757941 7546 tablet_bootstrap.cc:492] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612: Bootstrap starting.
I20260929 01:07:14.760383 7544 tablet_bootstrap.cc:492] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e: No bootstrap required, opened a new log
I20260929 01:07:14.760651 7544 ts_tablet_manager.cc:1397] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260929 01:07:14.761917 7546 tablet_bootstrap.cc:654] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:14.763083 7546 log.cc:824] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:14.764266 7543 raft_consensus.cc:348] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } }
I20260929 01:07:14.764703 7543 raft_consensus.cc:374] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:14.764822 7543 raft_consensus.cc:729] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc3ad18ebdb4d168516d49c00c7f0e6, State: Initialized, Role: FOLLOWER
I20260929 01:07:14.765479 7543 consensus_queue.cc:260] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } }
I20260929 01:07:14.766516 7385 heartbeater.cc:503] Master 127.5.198.254:44003 was elected leader, sending a full tablet report...
I20260929 01:07:14.766986 7543 ts_tablet_manager.cc:1428] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260929 01:07:14.767113 7546 tablet_bootstrap.cc:492] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612: No bootstrap required, opened a new log
I20260929 01:07:14.767361 7546 ts_tablet_manager.cc:1397] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260929 01:07:14.772810 7544 raft_consensus.cc:348] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } }
I20260929 01:07:14.773243 7544 raft_consensus.cc:374] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:14.773340 7544 raft_consensus.cc:729] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2cfeb4589b4e67b66d147c66bb0c0e, State: Initialized, Role: FOLLOWER
I20260929 01:07:14.773919 7544 consensus_queue.cc:260] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } }
I20260929 01:07:14.774981 7524 heartbeater.cc:503] Master 127.5.198.254:44003 was elected leader, sending a full tablet report...
I20260929 01:07:14.775656 7544 ts_tablet_manager.cc:1428] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260929 01:07:14.778141 7546 raft_consensus.cc:348] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } }
I20260929 01:07:14.778502 7546 raft_consensus.cc:374] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:14.778597 7546 raft_consensus.cc:729] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05ea8a14fa845dfbb729b0d89ef7612, State: Initialized, Role: FOLLOWER
I20260929 01:07:14.779179 7546 consensus_queue.cc:260] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } }
I20260929 01:07:14.780434 7546 ts_tablet_manager.cc:1428] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 01:07:14.808660 7550 raft_consensus.cc:482] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:07:14.809041 7550 raft_consensus.cc:504] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } }
I20260929 01:07:14.811322 7550 leader_election.cc:288] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbc3ad18ebdb4d168516d49c00c7f0e6 (127.5.198.194:45467), a05ea8a14fa845dfbb729b0d89ef7612 (127.5.198.193:34655)
I20260929 01:07:14.817399 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.811597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:45467 (local address 127.5.198.195:51603)
0929 01:07:14.811792 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:14.811811 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:14.811946 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:14.812556 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:14.812561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:14.812589 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:14.812769 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.812778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.814077 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.814083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.815602 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.815612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.816682 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.816688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.816862 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.816885 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 01:07:14.816982 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 01:07:14.817108 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 01:07:14.817399 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.811603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:51603 (local address 127.5.198.194:45467)
0929 01:07:14.811781 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:14.811785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:14.811953 (+ 168us) server_negotiation.cc:407] Connection header received
0929 01:07:14.812079 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:14.812085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:14.812161 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:14.812264 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:14.812915 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.813954 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.815751 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.816538 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.816693 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.816766 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:14.817003 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 01:07:14.817124 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:07:14.818430 7340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0b1d88e49c422db28a08e9a8ee8775" candidate_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" is_pre_election: true
I20260929 01:07:14.818614 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.812177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:58531 (local address 127.5.198.193:34655)
0929 01:07:14.812441 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:14.812450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:14.812906 (+ 456us) server_negotiation.cc:407] Connection header received
0929 01:07:14.813048 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:14.813052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:14.813115 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:14.813181 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:14.813771 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.814780 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.816550 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.817343 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.817545 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.817622 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:14.818251 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 01:07:14.818376 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:07:14.818672 7552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.812025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:34655 (local address 127.5.198.195:58531)
0929 01:07:14.812779 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:14.812793 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:14.812926 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:14.813226 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:14.813233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:14.813258 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:14.813616 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.813626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.814956 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.814961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.816341 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:14.816352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.817951 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.817957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:14.818093 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.818116 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 01:07:14.818224 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 01:07:14.818343 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":402,"threads_started":1}
I20260929 01:07:14.819022 7340 raft_consensus.cc:2454] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2cfeb4589b4e67b66d147c66bb0c0e in term 0.
I20260929 01:07:14.819675 7407 leader_election.cc:302] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2cfeb4589b4e67b66d147c66bb0c0e, fbc3ad18ebdb4d168516d49c00c7f0e6; no voters:
I20260929 01:07:14.819651 7196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0b1d88e49c422db28a08e9a8ee8775" candidate_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" is_pre_election: true
I20260929 01:07:14.820088 7550 raft_consensus.cc:2781] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:07:14.820106 7196 raft_consensus.cc:2454] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2cfeb4589b4e67b66d147c66bb0c0e in term 0.
I20260929 01:07:14.820194 7550 raft_consensus.cc:482] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:07:14.820308 7550 raft_consensus.cc:3037] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:14.823189 7550 raft_consensus.cc:504] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } }
I20260929 01:07:14.823781 7550 leader_election.cc:288] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [CANDIDATE]: Term 1 election: Requested vote from peers fbc3ad18ebdb4d168516d49c00c7f0e6 (127.5.198.194:45467), a05ea8a14fa845dfbb729b0d89ef7612 (127.5.198.193:34655)
I20260929 01:07:14.824237 7340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0b1d88e49c422db28a08e9a8ee8775" candidate_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6"
I20260929 01:07:14.824290 7196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0b1d88e49c422db28a08e9a8ee8775" candidate_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05ea8a14fa845dfbb729b0d89ef7612"
I20260929 01:07:14.824419 7340 raft_consensus.cc:3037] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:14.824572 7196 raft_consensus.cc:3037] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:14.826633 7340 raft_consensus.cc:2454] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2cfeb4589b4e67b66d147c66bb0c0e in term 1.
I20260929 01:07:14.826776 7196 raft_consensus.cc:2454] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2cfeb4589b4e67b66d147c66bb0c0e in term 1.
I20260929 01:07:14.827014 7407 leader_election.cc:302] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2cfeb4589b4e67b66d147c66bb0c0e, fbc3ad18ebdb4d168516d49c00c7f0e6; no voters:
I20260929 01:07:14.827272 7550 raft_consensus.cc:2781] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:14.827692 7550 raft_consensus.cc:686] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 1 LEADER]: Becoming Leader. State: Replica: 1c2cfeb4589b4e67b66d147c66bb0c0e, State: Running, Role: LEADER
I20260929 01:07:14.828145 7550 consensus_queue.cc:237] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } }
I20260929 01:07:14.832605 7039 catalog_manager.cc:5495] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2cfeb4589b4e67b66d147c66bb0c0e (127.5.198.195). New cstate: current_term: 1 leader_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } health_report { overall_health: UNKNOWN } } }
W20260929 01:07:14.872929 7247 tablet.cc:2369] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:07:14.877532 7386 tablet.cc:2369] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:07:14.882515 7540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:14.877119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.5.198.195:39053)
0929 01:07:14.877335 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:14.877343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:14.877385 (+ 42us) server_negotiation.cc:407] Connection header received
0929 01:07:14.877701 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:14.877707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:14.877812 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:14.877958 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:07:14.878317 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.879025 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.880317 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:14.880813 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:14.880954 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:14.881430 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:07:14.881894 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:07:14.882049 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:14.882167 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 01:07:14.882241 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 01:07:14.886888 7525 tablet.cc:2369] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:07:14.889943 7340 raft_consensus.cc:1261] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 1 FOLLOWER]: Refusing update from remote peer 1c2cfeb4589b4e67b66d147c66bb0c0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:14.889942 7196 raft_consensus.cc:1261] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612 [term 1 FOLLOWER]: Refusing update from remote peer 1c2cfeb4589b4e67b66d147c66bb0c0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:14.890782 7553 consensus_queue.cc:1035] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:14.891129 7550 consensus_queue.cc:1035] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:14.902359 7556 mvcc.cc:205] Tried to move back new op lower bound from 7334477966900633600 to 7334477966660341760. Current Snapshot: MvccSnapshot[applied={T|T < 7334477966900633600}]
I20260929 01:07:14.919919 7039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:07:14.920174 7039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:07:14.924060 7039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7: Sending DeleteTablet for 3 replicas of tablet af0b1d88e49c422db28a08e9a8ee8775
I20260929 01:07:14.925288 7176 tablet_service.cc:1514] Processing DeleteTablet for tablet af0b1d88e49c422db28a08e9a8ee8775 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:14 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:52578
I20260929 01:07:14.925731 7454 tablet_service.cc:1514] Processing DeleteTablet for tablet af0b1d88e49c422db28a08e9a8ee8775 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:14 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:34846
I20260929 01:07:14.925765 7315 tablet_service.cc:1514] Processing DeleteTablet for tablet af0b1d88e49c422db28a08e9a8ee8775 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:14 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:48566
I20260929 01:07:14.927567 7566 tablet_replica.cc:330] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612: stopping tablet replica
I20260929 01:07:14.927564 7039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 01:07:14.927999 7566 raft_consensus.cc:2229] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 01:07:14.928216 7039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:07:14.928544 7566 raft_consensus.cc:2258] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:14.941066 7568 tablet_replica.cc:330] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6: stopping tablet replica
I20260929 01:07:14.941592 7568 raft_consensus.cc:2229] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:07:14.941669 7176 tablet_service.cc:1467] Processing CreateTablet for tablet 99ee79970c904226b15d8373baaafac8 (DEFAULT_TABLE table=test-table [id=4b43094139c9422b9e0db69c16525d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:14.941679 7314 tablet_service.cc:1467] Processing CreateTablet for tablet 99ee79970c904226b15d8373baaafac8 (DEFAULT_TABLE table=test-table [id=4b43094139c9422b9e0db69c16525d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:14.942140 7176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ee79970c904226b15d8373baaafac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:14.942183 7568 raft_consensus.cc:2258] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:14.942324 7314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ee79970c904226b15d8373baaafac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:14.945554 7568 ts_tablet_manager.cc:1905] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:14.945843 7566 ts_tablet_manager.cc:1905] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:14.948743 7546 tablet_bootstrap.cc:492] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612: Bootstrap starting.
I20260929 01:07:14.949889 7543 tablet_bootstrap.cc:492] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6: Bootstrap starting.
I20260929 01:07:14.950484 7546 tablet_bootstrap.cc:654] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:14.951524 7543 tablet_bootstrap.cc:654] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:14.957696 7453 tablet_service.cc:1467] Processing CreateTablet for tablet 99ee79970c904226b15d8373baaafac8 (DEFAULT_TABLE table=test-table [id=4b43094139c9422b9e0db69c16525d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:14.958419 7453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ee79970c904226b15d8373baaafac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:14.959966 7568 ts_tablet_manager.cc:1918] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:07:14.962527 7566 ts_tablet_manager.cc:1918] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:07:14.962822 7566 log.cc:1196] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/af0b1d88e49c422db28a08e9a8ee8775
I20260929 01:07:14.962991 7567 tablet_replica.cc:330] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e: stopping tablet replica
I20260929 01:07:14.963830 7568 log.cc:1196] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/af0b1d88e49c422db28a08e9a8ee8775
I20260929 01:07:14.963915 7567 raft_consensus.cc:2229] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:07:14.964416 7566 ts_tablet_manager.cc:1939] T af0b1d88e49c422db28a08e9a8ee8775 P a05ea8a14fa845dfbb729b0d89ef7612: Deleting consensus metadata
I20260929 01:07:14.964648 7568 ts_tablet_manager.cc:1939] T af0b1d88e49c422db28a08e9a8ee8775 P fbc3ad18ebdb4d168516d49c00c7f0e6: Deleting consensus metadata
I20260929 01:07:14.964777 7567 raft_consensus.cc:2258] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:14.965874 7544 tablet_bootstrap.cc:492] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Bootstrap starting.
I20260929 01:07:14.967617 7544 tablet_bootstrap.cc:654] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:14.967780 7025 catalog_manager.cc:4841] TS fbc3ad18ebdb4d168516d49c00c7f0e6 (127.5.198.194:45467): tablet af0b1d88e49c422db28a08e9a8ee8775 (table test-table [id=7808d90b84ac4a46986fa67b7b81400b]) successfully deleted
I20260929 01:07:14.967527 7567 ts_tablet_manager.cc:1905] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:14.969062 7543 tablet_bootstrap.cc:492] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6: No bootstrap required, opened a new log
I20260929 01:07:14.969264 7543 ts_tablet_manager.cc:1397] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.004s
I20260929 01:07:14.970448 7543 raft_consensus.cc:348] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:14.970752 7543 raft_consensus.cc:374] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:14.970894 7543 raft_consensus.cc:729] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc3ad18ebdb4d168516d49c00c7f0e6, State: Initialized, Role: FOLLOWER
I20260929 01:07:14.971252 7543 consensus_queue.cc:260] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:14.972838 7543 ts_tablet_manager.cc:1428] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 01:07:14.974072 7546 tablet_bootstrap.cc:492] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612: No bootstrap required, opened a new log
I20260929 01:07:14.974277 7546 ts_tablet_manager.cc:1397] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.015s
I20260929 01:07:14.975154 7025 catalog_manager.cc:4841] TS a05ea8a14fa845dfbb729b0d89ef7612 (127.5.198.193:34655): tablet af0b1d88e49c422db28a08e9a8ee8775 (table test-table [id=7808d90b84ac4a46986fa67b7b81400b]) successfully deleted
I20260929 01:07:14.975183 7546 raft_consensus.cc:348] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:14.975514 7546 raft_consensus.cc:374] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:14.975648 7546 raft_consensus.cc:729] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05ea8a14fa845dfbb729b0d89ef7612, State: Initialized, Role: FOLLOWER
I20260929 01:07:14.975975 7546 consensus_queue.cc:260] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:14.976809 7546 ts_tablet_manager.cc:1428] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 01:07:14.977878 7544 tablet_bootstrap.cc:492] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e: No bootstrap required, opened a new log
I20260929 01:07:14.978049 7544 ts_tablet_manager.cc:1397] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.001s
I20260929 01:07:14.978434 7567 ts_tablet_manager.cc:1918] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:07:14.978657 7567 log.cc:1196] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/af0b1d88e49c422db28a08e9a8ee8775
I20260929 01:07:14.978940 7544 raft_consensus.cc:348] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:14.979173 7544 raft_consensus.cc:374] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:14.979293 7544 raft_consensus.cc:729] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2cfeb4589b4e67b66d147c66bb0c0e, State: Initialized, Role: FOLLOWER
I20260929 01:07:14.979414 7567 ts_tablet_manager.cc:1939] T af0b1d88e49c422db28a08e9a8ee8775 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Deleting consensus metadata
I20260929 01:07:14.980052 7544 consensus_queue.cc:260] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:14.980711 7544 ts_tablet_manager.cc:1428] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 01:07:14.981472 7026 catalog_manager.cc:4841] TS 1c2cfeb4589b4e67b66d147c66bb0c0e (127.5.198.195:39053): tablet af0b1d88e49c422db28a08e9a8ee8775 (table test-table [id=7808d90b84ac4a46986fa67b7b81400b]) successfully deleted
I20260929 01:07:15.041405 7551 raft_consensus.cc:482] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:07:15.041628 7551 raft_consensus.cc:504] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:15.043341 7551 leader_election.cc:288] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbc3ad18ebdb4d168516d49c00c7f0e6 (127.5.198.194:45467), 1c2cfeb4589b4e67b66d147c66bb0c0e (127.5.198.195:39053)
I20260929 01:07:15.052122 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:15.043834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:32841 (local address 127.5.198.194:45467)
0929 01:07:15.044071 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:15.044078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:15.045027 (+ 949us) server_negotiation.cc:407] Connection header received
0929 01:07:15.045241 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:15.045247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:15.045317 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:15.045406 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:15.045957 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.046659 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.049896 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.051179 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.051349 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:15.051424 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:15.051690 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 01:07:15.051816 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 01:07:15.052220 7572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:15.043683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:45467 (local address 127.5.198.193:32841)
0929 01:07:15.044932 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:15.044945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:15.045080 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:15.045445 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:15.045454 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:15.045487 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:15.045806 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:15.045813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.046787 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.046793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:15.049689 (+ 2896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:15.049701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.051396 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.051401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:15.051523 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:15.051546 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 01:07:15.051654 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 01:07:15.051782 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":8,"thread_start_us":377,"threads_started":1}
I20260929 01:07:15.053576 7340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ee79970c904226b15d8373baaafac8" candidate_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" is_pre_election: true
I20260929 01:07:15.053855 7340 raft_consensus.cc:2454] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a05ea8a14fa845dfbb729b0d89ef7612 in term 0.
I20260929 01:07:15.053930 7540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:15.044259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:51481 (local address 127.5.198.195:39053)
0929 01:07:15.044414 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:15.044419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:15.044535 (+ 116us) server_negotiation.cc:407] Connection header received
0929 01:07:15.044837 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:15.044843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:15.044941 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:15.045043 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:15.045667 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.046818 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.051513 (+ 4695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.052879 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.053037 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:15.053107 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:15.053465 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 01:07:15.053646 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:07:15.054414 7573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:15.043625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:39053 (local address 127.5.198.193:51481)
0929 01:07:15.044386 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:15.044409 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:15.044662 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:15.045122 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:15.045130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:15.045154 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:15.045500 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:15.045512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.046953 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.046960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:15.049687 (+ 2727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:15.049701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.053096 (+ 3395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.053101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:15.053291 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:15.053320 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 01:07:15.053436 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 01:07:15.053562 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":34,"thread_start_us":794,"threads_started":1}
I20260929 01:07:15.054435 7128 leader_election.cc:302] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a05ea8a14fa845dfbb729b0d89ef7612, fbc3ad18ebdb4d168516d49c00c7f0e6; no voters:
I20260929 01:07:15.054920 7551 raft_consensus.cc:2781] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:07:15.055071 7551 raft_consensus.cc:482] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:07:15.055173 7551 raft_consensus.cc:3037] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:15.055364 7474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ee79970c904226b15d8373baaafac8" candidate_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" is_pre_election: true
I20260929 01:07:15.055783 7474 raft_consensus.cc:2454] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a05ea8a14fa845dfbb729b0d89ef7612 in term 0.
I20260929 01:07:15.057736 7551 raft_consensus.cc:504] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:15.058367 7551 leader_election.cc:288] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [CANDIDATE]: Term 1 election: Requested vote from peers fbc3ad18ebdb4d168516d49c00c7f0e6 (127.5.198.194:45467), 1c2cfeb4589b4e67b66d147c66bb0c0e (127.5.198.195:39053)
I20260929 01:07:15.058748 7340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ee79970c904226b15d8373baaafac8" candidate_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6"
I20260929 01:07:15.058884 7474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ee79970c904226b15d8373baaafac8" candidate_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e"
I20260929 01:07:15.058971 7340 raft_consensus.cc:3037] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:15.059092 7474 raft_consensus.cc:3037] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:15.062391 7340 raft_consensus.cc:2454] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a05ea8a14fa845dfbb729b0d89ef7612 in term 1.
I20260929 01:07:15.062702 7474 raft_consensus.cc:2454] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a05ea8a14fa845dfbb729b0d89ef7612 in term 1.
I20260929 01:07:15.062819 7128 leader_election.cc:302] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a05ea8a14fa845dfbb729b0d89ef7612, fbc3ad18ebdb4d168516d49c00c7f0e6; no voters:
I20260929 01:07:15.063103 7551 raft_consensus.cc:2781] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:15.063625 7551 raft_consensus.cc:686] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [term 1 LEADER]: Becoming Leader. State: Replica: a05ea8a14fa845dfbb729b0d89ef7612, State: Running, Role: LEADER
I20260929 01:07:15.064079 7551 consensus_queue.cc:237] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:15.067540 7038 catalog_manager.cc:5495] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 reported cstate change: term changed from 0 to 1, leader changed from <none> to a05ea8a14fa845dfbb729b0d89ef7612 (127.5.198.193). New cstate: current_term: 1 leader_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } health_report { overall_health: UNKNOWN } } }
Sep 29 01:07:15 dist-test-slave-41cl krb5kdc[6994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644032, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:15.115139 7581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:15.099102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44003 (local address 127.0.0.1:47926)
0929 01:07:15.099777 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:15.099808 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:15.099959 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:15.100339 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:15.100344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:15.101273 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:15.101622 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:15.101634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.102719 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.102725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:15.104667 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:15.104682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.105425 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.105431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:15.105605 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:15.106389 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:15.106411 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:15.109195 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:15.112756 (+ 3561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:15.112765 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:15.112771 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:15.113173 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:15.113581 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:15.113585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:15.113587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:15.113722 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:15.114320 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:15.114326 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:15.114480 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 01:07:15.114615 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 01:07:15.114808 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":346,"threads_started":1}
I20260929 01:07:15.115396 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:15.099069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.5.198.254:44003)
0929 01:07:15.099282 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:15.099292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:15.099902 (+ 610us) server_negotiation.cc:407] Connection header received
0929 01:07:15.100108 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:15.100112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:15.100186 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:15.100274 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:15.101768 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.102588 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.104802 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.105251 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.105348 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:15.109374 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:15.109379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:15.109382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:15.109410 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:15.112612 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:15.113303 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:15.113308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:15.113312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:15.113465 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:15.113839 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:15.113844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:15.113849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:15.114182 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:15.114337 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:15.114753 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 01:07:15.114976 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:07:15.120772 7038 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:47926:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 01:07:15.121886 7038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:07:15.129478 7176 tablet_service.cc:1467] Processing CreateTablet for tablet 4070a4c269d44948b82409c594345b52 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a485602a52b4457694d609b4bd7e6bbc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:07:15.129505 7314 tablet_service.cc:1467] Processing CreateTablet for tablet 4070a4c269d44948b82409c594345b52 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a485602a52b4457694d609b4bd7e6bbc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:07:15.129992 7176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4070a4c269d44948b82409c594345b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:15.130007 7314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4070a4c269d44948b82409c594345b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:15.133889 7453 tablet_service.cc:1467] Processing CreateTablet for tablet 4070a4c269d44948b82409c594345b52 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a485602a52b4457694d609b4bd7e6bbc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:07:15.134388 7453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4070a4c269d44948b82409c594345b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:15.136739 7546 tablet_bootstrap.cc:492] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612: Bootstrap starting.
I20260929 01:07:15.138443 7546 tablet_bootstrap.cc:654] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:15.144620 7543 tablet_bootstrap.cc:492] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6: Bootstrap starting.
I20260929 01:07:15.146206 7544 tablet_bootstrap.cc:492] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Bootstrap starting.
I20260929 01:07:15.146276 7543 tablet_bootstrap.cc:654] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:15.147954 7544 tablet_bootstrap.cc:654] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:15.148175 7546 tablet_bootstrap.cc:492] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612: No bootstrap required, opened a new log
I20260929 01:07:15.148428 7546 ts_tablet_manager.cc:1397] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260929 01:07:15.149323 7546 raft_consensus.cc:348] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:15.149619 7546 raft_consensus.cc:374] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:15.149724 7546 raft_consensus.cc:729] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05ea8a14fa845dfbb729b0d89ef7612, State: Initialized, Role: FOLLOWER
I20260929 01:07:15.149993 7546 consensus_queue.cc:260] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:15.150445 7551 tablet_replica.cc:439] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } }
I20260929 01:07:15.150717 7551 tablet_replica.cc:442] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:07:15.152140 7546 ts_tablet_manager.cc:1428] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 01:07:15.155310 7543 tablet_bootstrap.cc:492] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6: No bootstrap required, opened a new log
I20260929 01:07:15.155532 7543 ts_tablet_manager.cc:1397] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20260929 01:07:15.155565 7544 tablet_bootstrap.cc:492] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e: No bootstrap required, opened a new log
I20260929 01:07:15.155737 7544 ts_tablet_manager.cc:1397] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 01:07:15.156265 7543 raft_consensus.cc:348] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:15.156548 7543 raft_consensus.cc:374] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:15.156672 7543 raft_consensus.cc:729] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc3ad18ebdb4d168516d49c00c7f0e6, State: Initialized, Role: FOLLOWER
I20260929 01:07:15.156550 7544 raft_consensus.cc:348] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:15.156797 7544 raft_consensus.cc:374] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:15.156880 7544 raft_consensus.cc:729] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2cfeb4589b4e67b66d147c66bb0c0e, State: Initialized, Role: FOLLOWER
I20260929 01:07:15.156965 7543 consensus_queue.cc:260] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:15.157121 7544 consensus_queue.cc:260] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:15.157418 7549 tablet_replica.cc:439] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } }
I20260929 01:07:15.157697 7544 ts_tablet_manager.cc:1428] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 01:07:15.157775 7549 tablet_replica.cc:442] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:07:15.158200 7543 ts_tablet_manager.cc:1428] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 01:07:15.158509 7550 tablet_replica.cc:439] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } }
I20260929 01:07:15.160828 7550 tablet_replica.cc:442] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:07:15.183449 7551 raft_consensus.cc:482] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:07:15.183647 7551 raft_consensus.cc:504] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:15.184257 7551 leader_election.cc:288] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbc3ad18ebdb4d168516d49c00c7f0e6 (127.5.198.194:45467), 1c2cfeb4589b4e67b66d147c66bb0c0e (127.5.198.195:39053)
I20260929 01:07:15.185335 7340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4070a4c269d44948b82409c594345b52" candidate_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" is_pre_election: true
I20260929 01:07:15.185568 7340 raft_consensus.cc:2454] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a05ea8a14fa845dfbb729b0d89ef7612 in term 0.
I20260929 01:07:15.185714 7474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4070a4c269d44948b82409c594345b52" candidate_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" is_pre_election: true
I20260929 01:07:15.185957 7474 raft_consensus.cc:2454] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a05ea8a14fa845dfbb729b0d89ef7612 in term 0.
I20260929 01:07:15.186019 7128 leader_election.cc:302] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a05ea8a14fa845dfbb729b0d89ef7612, fbc3ad18ebdb4d168516d49c00c7f0e6; no voters:
I20260929 01:07:15.186524 7551 raft_consensus.cc:2781] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:07:15.186623 7551 raft_consensus.cc:482] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:07:15.186724 7551 raft_consensus.cc:3037] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:15.189487 7551 raft_consensus.cc:504] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:15.190407 7551 leader_election.cc:288] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [CANDIDATE]: Term 1 election: Requested vote from peers fbc3ad18ebdb4d168516d49c00c7f0e6 (127.5.198.194:45467), 1c2cfeb4589b4e67b66d147c66bb0c0e (127.5.198.195:39053)
I20260929 01:07:15.190477 7474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4070a4c269d44948b82409c594345b52" candidate_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e"
I20260929 01:07:15.190464 7340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4070a4c269d44948b82409c594345b52" candidate_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6"
I20260929 01:07:15.190678 7340 raft_consensus.cc:3037] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:15.190685 7474 raft_consensus.cc:3037] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:15.193653 7340 raft_consensus.cc:2454] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a05ea8a14fa845dfbb729b0d89ef7612 in term 1.
I20260929 01:07:15.193898 7474 raft_consensus.cc:2454] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a05ea8a14fa845dfbb729b0d89ef7612 in term 1.
I20260929 01:07:15.194113 7128 leader_election.cc:302] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a05ea8a14fa845dfbb729b0d89ef7612, fbc3ad18ebdb4d168516d49c00c7f0e6; no voters:
I20260929 01:07:15.194371 7551 raft_consensus.cc:2781] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:15.194557 7551 raft_consensus.cc:686] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [term 1 LEADER]: Becoming Leader. State: Replica: a05ea8a14fa845dfbb729b0d89ef7612, State: Running, Role: LEADER
I20260929 01:07:15.194823 7551 consensus_queue.cc:237] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } }
I20260929 01:07:15.196923 7574 tablet_replica.cc:439] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612: TxnStatusTablet state changed. Reason: New leader a05ea8a14fa845dfbb729b0d89ef7612. Latest consensus state: current_term: 1 leader_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } }
I20260929 01:07:15.197279 7574 tablet_replica.cc:442] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:07:15.198725 7038 catalog_manager.cc:5495] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 reported cstate change: term changed from 0 to 1, leader changed from <none> to a05ea8a14fa845dfbb729b0d89ef7612 (127.5.198.193). New cstate: current_term: 1 leader_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } health_report { overall_health: UNKNOWN } } }
I20260929 01:07:15.200912 7586 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 01:07:15.201143 7586 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 01:07:15.223332 7581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:15.217602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:34655 (local address 127.0.0.1:52592)
0929 01:07:15.217846 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:15.217862 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:15.217966 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:15.218497 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:15.218502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:15.218513 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:07:15.218691 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:15.218699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.219747 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.219754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:15.220831 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:15.220841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.221549 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.221556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:15.221746 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:15.221795 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:07:15.222782 (+ 987us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:07:15.222802 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 01:07:15.222884 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 01:07:15.223037 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 01:07:15.223331 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:15.217617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52592 (local address 127.5.198.193:34655)
0929 01:07:15.218172 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:15.218177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:15.218191 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:07:15.218257 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:15.218260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:15.218332 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:15.218478 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:07:15.218815 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.219623 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.220965 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.221359 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.221533 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:15.222037 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:07:15.222634 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:07:15.222822 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:15.222903 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 01:07:15.222988 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452}
I20260929 01:07:15.227299 7340 raft_consensus.cc:1261] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 1 FOLLOWER]: Refusing update from remote peer a05ea8a14fa845dfbb729b0d89ef7612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:15.227605 7474 raft_consensus.cc:1261] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 1 FOLLOWER]: Refusing update from remote peer a05ea8a14fa845dfbb729b0d89ef7612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:15.228612 7551 consensus_queue.cc:1035] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:15.228961 7574 consensus_queue.cc:1035] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:15.233501 7549 tablet_replica.cc:439] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6: TxnStatusTablet state changed. Reason: New leader a05ea8a14fa845dfbb729b0d89ef7612. Latest consensus state: current_term: 1 leader_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } }
I20260929 01:07:15.233765 7549 tablet_replica.cc:442] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:07:15.233908 7551 tablet_replica.cc:439] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } }
I20260929 01:07:15.234166 7551 tablet_replica.cc:442] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:07:15.237506 7558 mvcc.cc:205] Tried to move back new op lower bound from 7334477968283172864 to 7334477968160419840. Current Snapshot: MvccSnapshot[applied={T|T < 7334477968283172864 or (T in {7334477968283172864})}]
I20260929 01:07:15.239102 7557 mvcc.cc:205] Tried to move back new op lower bound from 7334477968283172864 to 7334477968160419840. Current Snapshot: MvccSnapshot[applied={T|T < 7334477968283172864}]
I20260929 01:07:15.239372 7549 tablet_replica.cc:439] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } }
I20260929 01:07:15.239573 7549 tablet_replica.cc:442] T 4070a4c269d44948b82409c594345b52 P fbc3ad18ebdb4d168516d49c00c7f0e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:07:15.245579 7551 tablet_replica.cc:439] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } }
I20260929 01:07:15.245541 7550 tablet_replica.cc:439] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e: TxnStatusTablet state changed. Reason: New leader a05ea8a14fa845dfbb729b0d89ef7612. Latest consensus state: current_term: 1 leader_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } }
I20260929 01:07:15.245812 7550 tablet_replica.cc:442] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:07:15.245815 7551 tablet_replica.cc:442] T 4070a4c269d44948b82409c594345b52 P a05ea8a14fa845dfbb729b0d89ef7612: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:07:15.247855 7550 tablet_replica.cc:439] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05ea8a14fa845dfbb729b0d89ef7612" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 34655 } } peers { permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 } } peers { permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 } } }
I20260929 01:07:15.248093 7550 tablet_replica.cc:442] T 4070a4c269d44948b82409c594345b52 P 1c2cfeb4589b4e67b66d147c66bb0c0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:07:15.254662 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:15.243362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.5.198.193:34655)
0929 01:07:15.243785 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:15.243791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:15.243809 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:07:15.243947 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:15.243952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:15.244021 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:15.244142 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:07:15.249287 (+ 5145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.250717 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.252546 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.253099 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.253248 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:15.253959 (+ 711us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:07:15.254066 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:07:15.254162 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:15.254279 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 01:07:15.254350 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
I20260929 01:07:15.270304 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:15.264193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:60515 (local address 127.5.198.193:34655)
0929 01:07:15.264361 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:15.264369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:15.265038 (+ 669us) server_negotiation.cc:407] Connection header received
0929 01:07:15.265231 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:15.265239 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:15.265307 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:15.265392 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:07:15.265883 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.266618 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.268033 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.268963 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.269124 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:15.269507 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:07:15.269680 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:07:15.269761 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:15.269905 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 01:07:15.269972 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:07:15.270533 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:15.264264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:34655 (local address 127.5.198.193:60515)
0929 01:07:15.264940 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:15.264961 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:15.265108 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:15.265413 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:15.265417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:15.265427 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:07:15.265754 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:15.265762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.266746 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.266752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:15.267872 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:15.267883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.269164 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.269173 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:15.269297 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:15.269342 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:07:15.269772 (+ 430us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:07:15.269787 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 01:07:15.269890 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 01:07:15.270030 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":3,"thread_start_us":375,"threads_started":1}
I20260929 01:07:15.281107 7474 raft_consensus.cc:1261] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 1 FOLLOWER]: Refusing update from remote peer a05ea8a14fa845dfbb729b0d89ef7612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:15.282104 7551 consensus_queue.cc:1035] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2cfeb4589b4e67b66d147c66bb0c0e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:15.288708 7340 raft_consensus.cc:1261] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 1 FOLLOWER]: Refusing update from remote peer a05ea8a14fa845dfbb729b0d89ef7612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:15.289383 7574 consensus_queue.cc:1035] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbc3ad18ebdb4d168516d49c00c7f0e6" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:15.371802 7039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:07:15.372076 7039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:07:15.375142 7039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99709fd2347d4185b4ef5b3a597fd2c7: Sending DeleteTablet for 3 replicas of tablet 99ee79970c904226b15d8373baaafac8
I20260929 01:07:15.375983 7175 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ee79970c904226b15d8373baaafac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:15 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:52578
I20260929 01:07:15.376173 7314 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ee79970c904226b15d8373baaafac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:15 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:48566
I20260929 01:07:15.376405 7566 tablet_replica.cc:330] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612: stopping tablet replica
I20260929 01:07:15.376437 7568 tablet_replica.cc:330] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6: stopping tablet replica
I20260929 01:07:15.376624 7566 raft_consensus.cc:2229] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:07:15.376680 7568 raft_consensus.cc:2229] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:07:15.376878 7568 raft_consensus.cc:2258] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:15.376962 7566 raft_consensus.cc:2258] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:15.377602 7453 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ee79970c904226b15d8373baaafac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:15 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:34846
I20260929 01:07:15.377888 7567 tablet_replica.cc:330] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e: stopping tablet replica
I20260929 01:07:15.378022 7568 ts_tablet_manager.cc:1905] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:15.378098 7567 raft_consensus.cc:2229] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:07:15.378309 7567 raft_consensus.cc:2258] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:15.379289 7566 ts_tablet_manager.cc:1905] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:15.384902 7568 ts_tablet_manager.cc:1918] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:07:15.385061 7568 log.cc:1196] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/99ee79970c904226b15d8373baaafac8
I20260929 01:07:15.385483 7568 ts_tablet_manager.cc:1939] T 99ee79970c904226b15d8373baaafac8 P fbc3ad18ebdb4d168516d49c00c7f0e6: Deleting consensus metadata
I20260929 01:07:15.386675 7025 catalog_manager.cc:4841] TS fbc3ad18ebdb4d168516d49c00c7f0e6 (127.5.198.194:45467): tablet 99ee79970c904226b15d8373baaafac8 (table test-table [id=4b43094139c9422b9e0db69c16525d2b]) successfully deleted
I20260929 01:07:15.389034 7566 ts_tablet_manager.cc:1918] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:07:15.389300 7566 log.cc:1196] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/99ee79970c904226b15d8373baaafac8
I20260929 01:07:15.389601 7567 ts_tablet_manager.cc:1905] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:15.389935 7566 ts_tablet_manager.cc:1939] T 99ee79970c904226b15d8373baaafac8 P a05ea8a14fa845dfbb729b0d89ef7612: Deleting consensus metadata
I20260929 01:07:15.391269 7025 catalog_manager.cc:4841] TS a05ea8a14fa845dfbb729b0d89ef7612 (127.5.198.193:34655): tablet 99ee79970c904226b15d8373baaafac8 (table test-table [id=4b43094139c9422b9e0db69c16525d2b]) successfully deleted
I20260929 01:07:15.397166 7567 ts_tablet_manager.cc:1918] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:07:15.397339 7567 log.cc:1196] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/99ee79970c904226b15d8373baaafac8
I20260929 01:07:15.397791 7567 ts_tablet_manager.cc:1939] T 99ee79970c904226b15d8373baaafac8 P 1c2cfeb4589b4e67b66d147c66bb0c0e: Deleting consensus metadata
I20260929 01:07:15.398859 7026 catalog_manager.cc:4841] TS 1c2cfeb4589b4e67b66d147c66bb0c0e (127.5.198.195:39053): tablet 99ee79970c904226b15d8373baaafac8 (table test-table [id=4b43094139c9422b9e0db69c16525d2b]) successfully deleted
Sep 29 01:07:15 dist-test-slave-41cl krb5kdc[6994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644034, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:07:15.420161 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:15.396350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.5.198.193:34655)
0929 01:07:15.396584 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:15.396590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:15.399675 (+ 3085us) server_negotiation.cc:407] Connection header received
0929 01:07:15.399881 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:15.399886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:15.399985 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:15.400105 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:15.402411 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.403794 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.405711 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.406360 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.406494 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:15.412318 (+ 5824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:15.412336 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:15.412365 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:15.412438 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:15.417041 (+ 4603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:15.417900 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:15.417910 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:15.417914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:15.418054 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:15.418493 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:15.418503 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:15.418507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:15.418942 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:15.419242 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:15.419544 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 01:07:15.419776 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 01:07:15.421097 7241 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52602
I20260929 01:07:15.445276 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:15.424850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47940 (local address 127.5.198.254:44003)
0929 01:07:15.425064 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:15.425070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:15.428972 (+ 3902us) server_negotiation.cc:407] Connection header received
0929 01:07:15.429120 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:15.429127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:15.429213 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:15.429328 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:15.430956 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.432477 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.434279 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.434956 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.435086 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:15.438037 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:15.438043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:15.438049 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:15.438087 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:15.442112 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:15.442921 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:15.442927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:15.442932 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:15.443092 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:15.443531 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:15.443537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:15.443541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:15.443890 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:15.444164 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:15.444653 (+ 489us) server_negotiation.cc:299] Negotiation successful
0929 01:07:15.444875 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260929 01:07:15.445815 7039 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47940
I20260929 01:07:15.448050 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 7110
I20260929 01:07:15.472074 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:15.449684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.5.198.193:34655)
0929 01:07:15.449878 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:15.449883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:15.456308 (+ 6425us) server_negotiation.cc:407] Connection header received
0929 01:07:15.456534 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:15.456539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:15.456620 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:15.456723 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:15.458201 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.459346 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.461112 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.461848 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.461984 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:15.465235 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:15.465247 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:15.465253 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:15.465291 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:15.468763 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:15.469592 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:15.469601 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:15.469605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:15.469769 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:15.470222 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:15.470227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:15.470230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:15.470585 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:15.471285 (+ 700us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:15.471425 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 01:07:15.471644 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 01:07:15.472791 7241 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52604
W20260929 01:07:15.473364 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 7110: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:07:15.473546 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 7110
I20260929 01:07:15.493374 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 7250
Sep 29 01:07:15 dist-test-slave-41cl krb5kdc[6994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644034, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:07:15.507691 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:15.494251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.5.198.194:45467)
0929 01:07:15.494460 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:15.494467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:15.494487 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:07:15.494553 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:15.494557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:15.494623 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:15.494732 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:15.495875 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.496655 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.497880 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.498298 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.498441 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:15.502469 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:15.502481 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:15.502496 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:15.502526 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:15.505443 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:15.506074 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:15.506082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:15.506084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:15.506201 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:15.506542 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:15.506545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:15.506547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:15.506855 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:15.507055 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:15.507288 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 01:07:15.507446 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 01:07:15.508384 7380 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48572
W20260929 01:07:15.509021 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 7250: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:07:15.509158 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 7250
I20260929 01:07:15.526841 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 7389
Sep 29 01:07:15 dist-test-slave-41cl krb5kdc[6994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644034, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:07:15.541549 7540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:15.527691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34860 (local address 127.5.198.195:39053)
0929 01:07:15.527891 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:15.527897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:15.527995 (+ 98us) server_negotiation.cc:407] Connection header received
0929 01:07:15.528228 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:15.528232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:15.528321 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:15.528439 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:15.529615 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.530306 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.531520 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:15.531922 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:15.532059 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:15.536025 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:15.536036 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:15.536078 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:15.536122 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:15.539008 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:15.539596 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:15.539602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:15.539605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:15.539725 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:15.540062 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:15.540066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:15.540068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:15.540630 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:15.540875 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:15.541167 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 01:07:15.541324 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 01:07:15.542165 7519 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34860
W20260929 01:07:15.542763 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 7389: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:07:15.542907 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 7389
I20260929 01:07:15.563876 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 7007
I20260929 01:07:15.565006 7079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:15.681196 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 7007
2026-09-29T01:07:15Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4421 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:07:15 dist-test-slave-41cl krb5kdc[7608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:07:15 dist-test-slave-41cl krb5kdc[7608](info): set up 2 sockets
Sep 29 01:07:15 dist-test-slave-41cl krb5kdc[7608](info): commencing operation
krb5kdc: starting...
W20260929 01:07:16.764611 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:07:16 dist-test-slave-41cl krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:07:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:07:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:07:16.948999 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:39779
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:39779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:17.258193 7621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:17.258452 7621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:17.258541 7621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:17.265576 7621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:17.265666 7621 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 01:07:17.265703 7621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:17.265762 7621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:17.265816 7621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:17.265874 7621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:07:17.275163 7621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:39779
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:39779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:17.276268 7621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:17.277560 7621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:17.288942 7629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:17.288957 7627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:17.288933 7626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:17.290268 7621 server_base.cc:1034] running on GCE node
I20260929 01:07:17.290974 7621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:17.292287 7621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:17.293524 7621 hybrid_clock.cc:648] HybridClock initialized: now 1790644037293459 us; error 52 us; skew 500 ppm
I20260929 01:07:17.293870 7621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:17 dist-test-slave-41cl krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644037, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:17.296833 7621 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:07:17.298120 7621 webserver.cc:460] Webserver started at http://127.5.198.254:38847/ using document root <none> and password file <none>
I20260929 01:07:17.298704 7621 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:17.298784 7621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:17.299031 7621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:17.301753 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "4a8cbc22e7344791a2556e99ef17a9a6"
format_stamp: "Formatted at 2026-09-29 01:07:17 on dist-test-slave-41cl"
server_key: "12b229378873a25f0a7d4ffe12834ae8"
server_key_iv: "9238a24a5a43977a86d9a3270a0eb080"
server_key_version: "encryptionkey@0"
I20260929 01:07:17.302402 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "4a8cbc22e7344791a2556e99ef17a9a6"
format_stamp: "Formatted at 2026-09-29 01:07:17 on dist-test-slave-41cl"
server_key: "12b229378873a25f0a7d4ffe12834ae8"
server_key_iv: "9238a24a5a43977a86d9a3270a0eb080"
server_key_version: "encryptionkey@0"
I20260929 01:07:17.307035 7621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 01:07:17.310267 7636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:17.311477 7621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:07:17.311606 7621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "4a8cbc22e7344791a2556e99ef17a9a6"
format_stamp: "Formatted at 2026-09-29 01:07:17 on dist-test-slave-41cl"
server_key: "12b229378873a25f0a7d4ffe12834ae8"
server_key_iv: "9238a24a5a43977a86d9a3270a0eb080"
server_key_version: "encryptionkey@0"
I20260929 01:07:17.311769 7621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:17.329289 7621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:17.333458 7621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:17.333675 7621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:17.354653 7621 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:39779
I20260929 01:07:17.354677 7697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:39779 every 8 connection(s)
I20260929 01:07:17.356009 7621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:07:17.359694 7698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:17.364828 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 7621
I20260929 01:07:17.365047 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:07:17.365659 5915 external_mini_cluster.cc:1428] Setting key 3898031da2598875205765d438a960c2
I20260929 01:07:17.370015 7698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6: Bootstrap starting.
I20260929 01:07:17.373539 7698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:17.374631 7698 log.cc:824] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:17.377576 7698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6: No bootstrap required, opened a new log
Sep 29 01:07:17 dist-test-slave-41cl krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:17.385713 7698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8cbc22e7344791a2556e99ef17a9a6" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39779 } }
I20260929 01:07:17.386097 7698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:17.386214 7698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a8cbc22e7344791a2556e99ef17a9a6, State: Initialized, Role: FOLLOWER
I20260929 01:07:17.386885 7698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8cbc22e7344791a2556e99ef17a9a6" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39779 } }
I20260929 01:07:17.387157 7698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:07:17.387290 7698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:07:17.387465 7698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:17.388321 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:17.367612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44474 (local address 127.5.198.254:39779)
0929 01:07:17.368574 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:17.368586 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:17.368624 (+ 38us) server_negotiation.cc:407] Connection header received
0929 01:07:17.370281 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:17.370302 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:17.370683 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:17.370975 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:17.372735 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:17.374085 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:17.375680 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:17.376379 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:17.376589 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:17.381125 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:17.381133 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:17.381149 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:17.381192 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:17.384151 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:17.384911 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:17.384931 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:17.384941 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:17.385099 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:17.385474 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:17.385478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:17.385480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:17.385977 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:17.386301 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:17.386734 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 01:07:17.387204 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":475,"threads_started":1}
I20260929 01:07:17.389840 7698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8cbc22e7344791a2556e99ef17a9a6" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39779 } }
I20260929 01:07:17.390319 7698 leader_election.cc:302] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a8cbc22e7344791a2556e99ef17a9a6; no voters:
I20260929 01:07:17.390942 7698 leader_election.cc:288] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:07:17.391055 7703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:17.391824 7703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [term 1 LEADER]: Becoming Leader. State: Replica: 4a8cbc22e7344791a2556e99ef17a9a6, State: Running, Role: LEADER
I20260929 01:07:17.392410 7703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8cbc22e7344791a2556e99ef17a9a6" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39779 } }
I20260929 01:07:17.393251 7698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:17.395097 7705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a8cbc22e7344791a2556e99ef17a9a6. Latest consensus state: current_term: 1 leader_uuid: "4a8cbc22e7344791a2556e99ef17a9a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8cbc22e7344791a2556e99ef17a9a6" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39779 } } }
I20260929 01:07:17.395428 7705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:17.400075 7704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a8cbc22e7344791a2556e99ef17a9a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8cbc22e7344791a2556e99ef17a9a6" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39779 } } }
I20260929 01:07:17.400403 7704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:17.401110 7708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:07:17.406226 7708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:07:17.414476 7708 catalog_manager.cc:1347] Generated new cluster ID: e9282c5cd1dd4798b86c7834358e7ce5
I20260929 01:07:17.414608 7708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:07:17.433288 7708 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:07:17.434512 7708 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:07:17.443316 7708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6: Generated new TSK 0
I20260929 01:07:17.444108 7708 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:07:17.500983 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:39779
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:17.819875 7724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:17.820168 7724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:17.820349 7724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:17.827411 7724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:17.827549 7724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:17.827710 7724 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 01:07:17.827804 7724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:07:17.827917 7724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:07:17.837548 7724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.198.254:39779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:17.838770 7724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:17.840044 7724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:17.852798 7729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:17.852891 7730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:17.852917 7732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:17.853884 7724 server_base.cc:1034] running on GCE node
I20260929 01:07:17.854401 7724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:17.855373 7724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:17.856632 7724 hybrid_clock.cc:648] HybridClock initialized: now 1790644037856571 us; error 44 us; skew 500 ppm
I20260929 01:07:17.857023 7724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:17 dist-test-slave-41cl krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644037, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:17.861191 7724 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:07:17.863938 7724 webserver.cc:460] Webserver started at http://127.5.198.193:40227/ using document root <none> and password file <none>
I20260929 01:07:17.864753 7724 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:17.864882 7724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:17.865238 7724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:17.867921 7724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "761f5e9739294980920bcdba74d00b9f"
format_stamp: "Formatted at 2026-09-29 01:07:17 on dist-test-slave-41cl"
server_key: "63cb0ae8dc1813744aeb1a2a11365dd6"
server_key_iv: "1ac90069d4d035e012dc6a30b51892d1"
server_key_version: "encryptionkey@0"
I20260929 01:07:17.868706 7724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "761f5e9739294980920bcdba74d00b9f"
format_stamp: "Formatted at 2026-09-29 01:07:17 on dist-test-slave-41cl"
server_key: "63cb0ae8dc1813744aeb1a2a11365dd6"
server_key_iv: "1ac90069d4d035e012dc6a30b51892d1"
server_key_version: "encryptionkey@0"
I20260929 01:07:17.873431 7724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:07:17.876721 7739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:17.878017 7724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:07:17.878176 7724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "761f5e9739294980920bcdba74d00b9f"
format_stamp: "Formatted at 2026-09-29 01:07:17 on dist-test-slave-41cl"
server_key: "63cb0ae8dc1813744aeb1a2a11365dd6"
server_key_iv: "1ac90069d4d035e012dc6a30b51892d1"
server_key_version: "encryptionkey@0"
I20260929 01:07:17.878353 7724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:17.909584 7724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:17.913480 7724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:17.913765 7724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:17.916551 7724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:17.916671 7724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:17.916802 7724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:17.916889 7724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:07:17 dist-test-slave-41cl krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644037, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:17.940842 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:17.917093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:46241 (local address 127.5.198.254:39779)
0929 01:07:17.917580 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:17.917588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:17.919201 (+ 1613us) server_negotiation.cc:407] Connection header received
0929 01:07:17.921962 (+ 2761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:17.921967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:17.922033 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:17.922097 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:17.924369 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:17.925736 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:17.927849 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:17.928643 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:17.928799 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:17.933370 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:17.933380 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:17.933388 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:17.933438 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:17.937318 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:17.938209 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:17.938217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:17.938224 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:17.938393 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:17.938802 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:17.938809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:17.938815 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:17.939223 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:17.939386 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:17.940023 (+ 637us) server_negotiation.cc:299] Negotiation successful
0929 01:07:17.940304 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":239,"threads_started":1}
I20260929 01:07:17.942544 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:17.918180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39779 (local address 127.5.198.193:46241)
0929 01:07:17.919032 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:17.919078 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:17.921578 (+ 2500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:17.922637 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:17.922645 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:17.923428 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:17.924144 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:17.924166 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:17.925908 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:17.925914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:17.927693 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:17.927711 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:17.928900 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:17.928910 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:17.929198 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:17.930113 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:17.930134 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:17.933143 (+ 3009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:17.937469 (+ 4326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:17.937484 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:17.937500 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:17.938096 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:17.938531 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:17.938534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:17.938537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:17.938696 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:17.939368 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:17.939373 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:17.939674 (+ 301us) client_negotiation.cc:769] Sending connection context
0929 01:07:17.939890 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 01:07:17.940225 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":382,"threads_started":1}
I20260929 01:07:17.961495 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:17.948215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:35911 (local address 127.5.198.254:39779)
0929 01:07:17.948355 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:17.948359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:17.948943 (+ 584us) server_negotiation.cc:407] Connection header received
0929 01:07:17.949151 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:17.949156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:17.949225 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:17.949321 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:17.950170 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:17.950847 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:17.952968 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:17.953547 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:17.953685 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:17.956056 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:17.956062 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:17.956066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:17.956125 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:17.958993 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:17.959604 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:17.959609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:17.959612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:17.959733 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:17.960073 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:17.960079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:17.960083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:17.960331 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:17.960484 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:17.960946 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 01:07:17.961150 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 01:07:17.961584 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:17.948236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39779 (local address 127.5.198.193:35911)
0929 01:07:17.948844 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:17.948862 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:17.948999 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:17.949342 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:17.949346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:17.949770 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:17.950041 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:17.950049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:17.950991 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:17.950996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:17.952817 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:17.952828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:17.953728 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:17.953734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:17.953931 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:17.954739 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:17.954758 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:17.955886 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:17.959120 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:17.959125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:17.959129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:17.959486 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:17.959838 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:17.959844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:17.959847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:17.959972 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:17.960447 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:17.960452 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:17.960675 (+ 223us) client_negotiation.cc:769] Sending connection context
0929 01:07:17.960834 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 01:07:17.961131 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":363,"threads_started":1}
I20260929 01:07:17.970839 7724 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:35493
I20260929 01:07:17.970876 7859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:35493 every 8 connection(s)
I20260929 01:07:17.971987 7724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:07:17.973835 7860 heartbeater.cc:347] Connected to a master server at 127.5.198.254:39779
I20260929 01:07:17.974078 7860 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:17.974649 7860 heartbeater.cc:511] Master 127.5.198.254:39779 requested a full tablet report, sending...
I20260929 01:07:17.976437 7653 ts_manager.cc:194] Registered new tserver with Master: 761f5e9739294980920bcdba74d00b9f (127.5.198.193:35493)
I20260929 01:07:17.977896 7653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:46241
I20260929 01:07:17.978973 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 7724
I20260929 01:07:17.979161 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:07:17.979799 5915 external_mini_cluster.cc:1428] Setting key 49e120c2f632395e60c130003b1c77fc
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:07:18.020620 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:39779
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:18.335332 7864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:18.335642 7864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:18.335820 7864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:18.342979 7864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:18.343134 7864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:18.343276 7864 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 01:07:18.343360 7864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:07:18.343494 7864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:07:18.353462 7864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.198.254:39779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:18.354779 7864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:18.356045 7864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:18.369150 7870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:18.369408 7872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:18.369671 7869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:18.370223 7864 server_base.cc:1034] running on GCE node
I20260929 01:07:18.370755 7864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:18.371594 7864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:18.372849 7864 hybrid_clock.cc:648] HybridClock initialized: now 1790644038372784 us; error 54 us; skew 500 ppm
I20260929 01:07:18.373152 7864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:18 dist-test-slave-41cl krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:18.376250 7864 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:07:18.377626 7864 webserver.cc:460] Webserver started at http://127.5.198.194:46323/ using document root <none> and password file <none>
I20260929 01:07:18.378218 7864 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:18.378314 7864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:18.378572 7864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:18.381361 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb"
format_stamp: "Formatted at 2026-09-29 01:07:18 on dist-test-slave-41cl"
server_key: "e8b51c048f614ed04372bef5caec67c4"
server_key_iv: "35ae3974ac3d36305718dfe3f304de33"
server_key_version: "encryptionkey@0"
I20260929 01:07:18.382015 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb"
format_stamp: "Formatted at 2026-09-29 01:07:18 on dist-test-slave-41cl"
server_key: "e8b51c048f614ed04372bef5caec67c4"
server_key_iv: "35ae3974ac3d36305718dfe3f304de33"
server_key_version: "encryptionkey@0"
I20260929 01:07:18.386857 7864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 01:07:18.390007 7879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:18.391191 7864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:07:18.391310 7864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb"
format_stamp: "Formatted at 2026-09-29 01:07:18 on dist-test-slave-41cl"
server_key: "e8b51c048f614ed04372bef5caec67c4"
server_key_iv: "35ae3974ac3d36305718dfe3f304de33"
server_key_version: "encryptionkey@0"
I20260929 01:07:18.391460 7864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:18.426470 7864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:18.430629 7864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:18.430873 7864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:18.433385 7864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:18.433475 7864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:18.433632 7864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:18.433725 7864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:07:18 dist-test-slave-41cl krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:18.458879 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:18.433771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:49601 (local address 127.5.198.254:39779)
0929 01:07:18.433935 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:18.433939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:18.435413 (+ 1474us) server_negotiation.cc:407] Connection header received
0929 01:07:18.437538 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:18.437546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:18.437654 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:18.437771 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:18.440442 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:18.441455 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:18.444166 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:18.444752 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:18.444890 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:18.450265 (+ 5375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:18.450274 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:18.450279 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:18.450322 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:18.454391 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:18.455335 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:18.455339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:18.455341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:18.455450 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:18.456118 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:18.456126 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:18.456133 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:18.456598 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:18.456760 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:18.458338 (+ 1578us) server_negotiation.cc:299] Negotiation successful
0929 01:07:18.458545 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 01:07:18.459725 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:18.434415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39779 (local address 127.5.198.194:49601)
0929 01:07:18.435236 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:18.435278 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:18.437257 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:18.438226 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:18.438241 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:18.439173 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:18.440207 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:18.440245 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:18.441633 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:18.441639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:18.443992 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:18.444011 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:18.445102 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:18.445112 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:18.445538 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:18.446384 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:18.446413 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:18.450008 (+ 3595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:18.454544 (+ 4536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:18.454550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:18.454563 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:18.455174 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:18.455611 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:18.455619 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:18.455626 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:18.455950 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:18.456854 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:18.456866 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:18.457446 (+ 580us) client_negotiation.cc:769] Sending connection context
0929 01:07:18.457838 (+ 392us) client_negotiation.cc:240] Negotiation successful
0929 01:07:18.458450 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":348,"threads_started":1}
I20260929 01:07:18.477011 7864 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:46633
I20260929 01:07:18.477011 7998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:46633 every 8 connection(s)
I20260929 01:07:18.478341 7864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:07:18.480703 7999 heartbeater.cc:347] Connected to a master server at 127.5.198.254:39779
I20260929 01:07:18.480934 7999 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:18.481729 7999 heartbeater.cc:511] Master 127.5.198.254:39779 requested a full tablet report, sending...
I20260929 01:07:18.482988 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:18.467885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39779 (local address 127.5.198.194:45179)
0929 01:07:18.468613 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:18.468631 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:18.468777 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:18.469091 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:18.469096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:18.469680 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:18.470228 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:18.470248 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:18.471896 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:18.471901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:18.474266 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:18.474281 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:18.475280 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:18.475284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:18.475391 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:18.476064 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:18.476093 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:18.476949 (+ 856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:18.480132 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:18.480137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:18.480140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:18.480665 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:18.481073 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:18.481084 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:18.481087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:18.481245 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:18.482014 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:18.482018 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:18.482196 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 01:07:18.482348 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 01:07:18.482648 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":417,"threads_started":1}
I20260929 01:07:18.483062 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:18.467735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:45179 (local address 127.5.198.254:39779)
0929 01:07:18.467909 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:18.467913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:18.468707 (+ 794us) server_negotiation.cc:407] Connection header received
0929 01:07:18.468902 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:18.468907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:18.468974 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:18.469068 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:18.470396 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:18.471693 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:18.474489 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:18.475112 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:18.475254 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:18.477144 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:18.477151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:18.477157 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:18.477193 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:18.480016 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:18.480795 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:18.480800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:18.480803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:18.480960 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:18.481387 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:18.481394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:18.481399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:18.481802 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:18.481944 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:18.482519 (+ 575us) server_negotiation.cc:299] Negotiation successful
0929 01:07:18.482761 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:07:18.483268 7653 ts_manager.cc:194] Registered new tserver with Master: 3e85bba98ca04bb3ae2e956a44aa4aeb (127.5.198.194:46633)
I20260929 01:07:18.484112 7653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:49601
I20260929 01:07:18.488145 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 7864
I20260929 01:07:18.488288 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:07:18.488742 5915 external_mini_cluster.cc:1428] Setting key c29f362ea54b64fa695894dfe0c64dee
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:07:18.527977 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:39779
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:39963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:18.851073 8003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:18.851347 8003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:18.851500 8003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:18.859411 8003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:18.859517 8003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:18.859663 8003 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 01:07:18.859738 8003 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:07:18.859843 8003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:07:18.869937 8003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:39963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.198.254:39779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:18.871230 8003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:18.872545 8003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:18.885910 8008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:18.886250 8009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:18.886509 8011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:18.887372 8003 server_base.cc:1034] running on GCE node
I20260929 01:07:18.887933 8003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:18.888974 8003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:18.890180 8003 hybrid_clock.cc:648] HybridClock initialized: now 1790644038890116 us; error 61 us; skew 500 ppm
I20260929 01:07:18.890467 8003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:18 dist-test-slave-41cl krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:18.893821 8003 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:07:18.895694 8003 webserver.cc:460] Webserver started at http://127.5.198.195:42991/ using document root <none> and password file <none>
I20260929 01:07:18.896433 8003 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:18.896613 8003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:18.896871 8003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:18.899662 8003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "b7b4408c7a2b42b7bd71232da487152a"
format_stamp: "Formatted at 2026-09-29 01:07:18 on dist-test-slave-41cl"
server_key: "a22e140c8b6af97cf0cd1b1ea5570118"
server_key_iv: "756c158801be08124358e41c95d43e1b"
server_key_version: "encryptionkey@0"
I20260929 01:07:18.900339 8003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "b7b4408c7a2b42b7bd71232da487152a"
format_stamp: "Formatted at 2026-09-29 01:07:18 on dist-test-slave-41cl"
server_key: "a22e140c8b6af97cf0cd1b1ea5570118"
server_key_iv: "756c158801be08124358e41c95d43e1b"
server_key_version: "encryptionkey@0"
I20260929 01:07:18.905735 8003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:07:18.909111 8018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:18.910575 8003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:07:18.910754 8003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "b7b4408c7a2b42b7bd71232da487152a"
format_stamp: "Formatted at 2026-09-29 01:07:18 on dist-test-slave-41cl"
server_key: "a22e140c8b6af97cf0cd1b1ea5570118"
server_key_iv: "756c158801be08124358e41c95d43e1b"
server_key_version: "encryptionkey@0"
I20260929 01:07:18.910964 8003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:18.925748 8003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:18.930114 8003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:18.930408 8003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:18.933094 8003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:18.933233 8003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:18.933365 8003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:18.933475 8003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:07:18 dist-test-slave-41cl krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644038, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:18.961156 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:18.933624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:38639 (local address 127.5.198.254:39779)
0929 01:07:18.933883 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:18.933892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:18.935351 (+ 1459us) server_negotiation.cc:407] Connection header received
0929 01:07:18.938634 (+ 3283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:18.938640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:18.938742 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:18.938826 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:18.941927 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:18.942945 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:18.945655 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:18.946411 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:18.946717 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:18.951664 (+ 4947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:18.951674 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:18.951682 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:18.951735 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:18.955479 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:18.956577 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:18.956585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:18.956591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:18.956748 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:18.957381 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:18.957387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:18.957390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:18.957768 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:18.957926 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:18.960717 (+ 2791us) server_negotiation.cc:299] Negotiation successful
0929 01:07:18.960920 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 01:07:18.961719 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:18.934298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39779 (local address 127.5.198.195:38639)
0929 01:07:18.935164 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:18.935217 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:18.938206 (+ 2989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:18.939498 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:18.939514 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:18.940596 (+ 1082us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:18.941680 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:18.941726 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:18.943168 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:18.943173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:18.945419 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:18.945501 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:18.946799 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:18.946809 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:18.947054 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:18.947940 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:18.947972 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:18.951363 (+ 3391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:18.955609 (+ 4246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:18.955618 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:18.955638 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:18.956334 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:18.956870 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:18.956877 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:18.956884 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:18.957213 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:18.957924 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:18.957936 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:18.958527 (+ 591us) client_negotiation.cc:769] Sending connection context
0929 01:07:18.958910 (+ 383us) client_negotiation.cc:240] Negotiation successful
0929 01:07:18.959695 (+ 785us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":345,"threads_started":1}
I20260929 01:07:18.980674 7860 heartbeater.cc:503] Master 127.5.198.254:39779 was elected leader, sending a full tablet report...
I20260929 01:07:18.984227 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:18.967186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:43709 (local address 127.5.198.254:39779)
0929 01:07:18.967332 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:18.967336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:18.967991 (+ 655us) server_negotiation.cc:407] Connection header received
0929 01:07:18.968179 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:18.968182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:18.968246 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:18.968343 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:18.969623 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:18.970958 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:18.974413 (+ 3455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:18.975017 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:18.975143 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:18.978011 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:18.978019 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:18.978026 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:18.978067 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:18.981511 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:18.982243 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:18.982248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:18.982251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:18.982401 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:18.982825 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:18.982830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:18.982834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:18.983097 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:18.983258 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:18.983638 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 01:07:18.983861 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 01:07:18.984315 8106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:18.967332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39779 (local address 127.5.198.195:43709)
0929 01:07:18.967894 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:18.967912 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:18.968058 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:18.968363 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:18.968375 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:18.968968 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:18.969427 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:18.969445 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:18.972591 (+ 3146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:18.972597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:18.974236 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:18.974254 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:18.975213 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:18.975219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:18.975445 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:18.976122 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:18.976146 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:18.977764 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:18.981650 (+ 3886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:18.981655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:18.981658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:18.982123 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:18.982534 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:18.982539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:18.982543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:18.982696 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:18.983213 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:18.983218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:18.983403 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 01:07:18.983534 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 01:07:18.983844 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":325,"threads_started":1}
I20260929 01:07:18.989672 8003 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:42389
I20260929 01:07:18.989694 8137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:42389 every 8 connection(s)
I20260929 01:07:18.990823 8003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:07:18.993139 8138 heartbeater.cc:347] Connected to a master server at 127.5.198.254:39779
I20260929 01:07:18.993395 8138 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:18.994185 8138 heartbeater.cc:511] Master 127.5.198.254:39779 requested a full tablet report, sending...
I20260929 01:07:18.995095 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 8003
I20260929 01:07:18.995291 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:07:18.995692 7653 ts_manager.cc:194] Registered new tserver with Master: b7b4408c7a2b42b7bd71232da487152a (127.5.198.195:42389)
I20260929 01:07:18.995810 5915 external_mini_cluster.cc:1428] Setting key 88043e26a140d356dae731348f7d2b32
I20260929 01:07:18.996821 7653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:38639
I20260929 01:07:18.999215 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 01:07:19 dist-test-slave-41cl krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 01:07:19 dist-test-slave-41cl krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:19.053201 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.032179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44484 (local address 127.5.198.254:39779)
0929 01:07:19.032357 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.032362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.032830 (+ 468us) server_negotiation.cc:407] Connection header received
0929 01:07:19.033000 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.033004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.033066 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.033156 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:19.034776 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.035612 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.036989 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.037403 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.037509 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.047710 (+ 10201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:19.047715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:19.047719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:19.047752 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:19.050712 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:19.051285 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:19.051289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:19.051291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:19.051415 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:19.051845 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:19.051849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:19.051851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:19.052114 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:19.052293 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.052692 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.052858 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:07:19.057268 7653 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:07:19.064137 7653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:07:19.089304 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.077107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:35493 (local address 127.0.0.1:50904)
0929 01:07:19.079376 (+ 2269us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:19.079420 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:19.079683 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:19.080772 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:19.080780 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:19.080814 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:19.082056 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.082073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.083902 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.083910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.086190 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.086201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.088037 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.088044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.088289 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.088348 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 01:07:19.088543 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 01:07:19.088757 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1994,"mutex_wait_us":729,"thread_start_us":1854,"threads_started":1}
I20260929 01:07:19.090265 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.077735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55006 (local address 127.5.198.195:42389)
0929 01:07:19.079034 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.079046 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.079689 (+ 643us) server_negotiation.cc:407] Connection header received
0929 01:07:19.079976 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.079992 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.080267 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.080504 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:19.081859 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.083641 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.087073 (+ 3432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.088167 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.088324 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.088576 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.089765 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.089938 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1075,"thread_start_us":827,"threads_started":1}
I20260929 01:07:19.089610 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.077500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:42389 (local address 127.0.0.1:55006)
0929 01:07:19.079550 (+ 2050us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:19.079571 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:19.079742 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:19.080830 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:19.080836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:19.080866 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:19.081518 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.081540 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.083831 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.083838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.086908 (+ 3070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.086919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.088793 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.088799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.088964 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.088996 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 01:07:19.089092 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 01:07:19.089218 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1931,"mutex_wait_us":129,"thread_start_us":1762,"threads_started":1}
I20260929 01:07:19.090629 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.077414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.5.198.193:35493)
0929 01:07:19.078072 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.078086 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.079530 (+ 1444us) server_negotiation.cc:407] Connection header received
0929 01:07:19.079940 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.079953 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.080277 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.080451 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:19.082215 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.083750 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.086336 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.087798 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.087940 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.088140 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.089893 (+ 1753us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.090101 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":711,"threads_started":1}
I20260929 01:07:19.091055 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.079769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:46633 (local address 127.0.0.1:44362)
0929 01:07:19.082534 (+ 2765us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:19.082552 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:19.082653 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:19.083771 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:19.083775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:19.083804 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:19.084436 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.084450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.086428 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.086433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.088441 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.088454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.090190 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.090197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.090406 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.090446 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 01:07:19.090542 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 01:07:19.090680 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2652,"thread_start_us":2990,"threads_started":1}
I20260929 01:07:19.091156 8156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.081027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.5.198.194:46633)
0929 01:07:19.083247 (+ 2220us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.083255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.083280 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:07:19.083370 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.083377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.083572 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.083762 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:19.084705 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.086224 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.088636 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.089973 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.090122 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.090314 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.090665 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.090903 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2078,"thread_start_us":1986,"threads_started":1}
I20260929 01:07:19.094293 7929 tablet_service.cc:1467] Processing CreateTablet for tablet 9a76e04bd895451495d2a1cae3af83f0 (DEFAULT_TABLE table=test-table [id=2d090ac7908e41339ace6f157a680cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:19.094309 8068 tablet_service.cc:1467] Processing CreateTablet for tablet 9a76e04bd895451495d2a1cae3af83f0 (DEFAULT_TABLE table=test-table [id=2d090ac7908e41339ace6f157a680cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:19.094305 7790 tablet_service.cc:1467] Processing CreateTablet for tablet 9a76e04bd895451495d2a1cae3af83f0 (DEFAULT_TABLE table=test-table [id=2d090ac7908e41339ace6f157a680cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:19.095690 7790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a76e04bd895451495d2a1cae3af83f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:19.095705 7929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a76e04bd895451495d2a1cae3af83f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:19.096333 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a76e04bd895451495d2a1cae3af83f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:19.106630 8157 tablet_bootstrap.cc:492] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Bootstrap starting.
I20260929 01:07:19.109957 8157 tablet_bootstrap.cc:654] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:19.110666 8158 tablet_bootstrap.cc:492] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f: Bootstrap starting.
I20260929 01:07:19.111413 8157 log.cc:824] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:19.112574 8159 tablet_bootstrap.cc:492] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a: Bootstrap starting.
I20260929 01:07:19.114640 8158 tablet_bootstrap.cc:654] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:19.115803 8159 tablet_bootstrap.cc:654] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:19.116194 8158 log.cc:824] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:19.117102 8159 log.cc:824] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:19.119320 8158 tablet_bootstrap.cc:492] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f: No bootstrap required, opened a new log
I20260929 01:07:19.119586 8158 ts_tablet_manager.cc:1397] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 01:07:19.119587 8157 tablet_bootstrap.cc:492] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb: No bootstrap required, opened a new log
I20260929 01:07:19.119908 8157 ts_tablet_manager.cc:1397] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260929 01:07:19.126426 8159 tablet_bootstrap.cc:492] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a: No bootstrap required, opened a new log
I20260929 01:07:19.126685 8159 ts_tablet_manager.cc:1397] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260929 01:07:19.127130 8158 raft_consensus.cc:348] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } }
I20260929 01:07:19.127491 8158 raft_consensus.cc:374] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:19.127589 8158 raft_consensus.cc:729] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761f5e9739294980920bcdba74d00b9f, State: Initialized, Role: FOLLOWER
I20260929 01:07:19.128235 8158 consensus_queue.cc:260] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } }
I20260929 01:07:19.130265 8158 ts_tablet_manager.cc:1428] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260929 01:07:19.131587 8157 raft_consensus.cc:348] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } }
I20260929 01:07:19.132018 8157 raft_consensus.cc:374] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:19.132124 8157 raft_consensus.cc:729] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e85bba98ca04bb3ae2e956a44aa4aeb, State: Initialized, Role: FOLLOWER
I20260929 01:07:19.132752 8157 consensus_queue.cc:260] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } }
I20260929 01:07:19.133873 7999 heartbeater.cc:503] Master 127.5.198.254:39779 was elected leader, sending a full tablet report...
I20260929 01:07:19.134493 8157 ts_tablet_manager.cc:1428] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260929 01:07:19.136993 8159 raft_consensus.cc:348] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } }
I20260929 01:07:19.137351 8159 raft_consensus.cc:374] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:19.137476 8159 raft_consensus.cc:729] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b4408c7a2b42b7bd71232da487152a, State: Initialized, Role: FOLLOWER
I20260929 01:07:19.138073 8159 consensus_queue.cc:260] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } }
I20260929 01:07:19.138998 8138 heartbeater.cc:503] Master 127.5.198.254:39779 was elected leader, sending a full tablet report...
I20260929 01:07:19.139478 8159 ts_tablet_manager.cc:1428] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a: Time spent starting tablet: real 0.013s user 0.005s sys 0.009s
I20260929 01:07:19.153553 8165 raft_consensus.cc:482] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:07:19.153852 8165 raft_consensus.cc:504] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } }
I20260929 01:07:19.155392 8165 leader_election.cc:288] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 761f5e9739294980920bcdba74d00b9f (127.5.198.193:35493), 3e85bba98ca04bb3ae2e956a44aa4aeb (127.5.198.194:46633)
I20260929 01:07:19.162231 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.155770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:35493 (local address 127.5.198.195:50723)
0929 01:07:19.155993 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:19.156013 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:19.156144 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:19.156969 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:19.156973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:19.157002 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:19.157182 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.157190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.158457 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.158461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.160259 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.160273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.161607 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.161611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.161809 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.161834 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 01:07:19.161950 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 01:07:19.162039 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 01:07:19.162647 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.155766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:50723 (local address 127.5.198.193:35493)
0929 01:07:19.155980 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.155984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.156089 (+ 105us) server_negotiation.cc:407] Connection header received
0929 01:07:19.156393 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.156398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.156507 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.156618 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:19.157337 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.158316 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.160417 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.161460 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.161601 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.161673 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.161982 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.162110 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 01:07:19.163184 8156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.156564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:48501 (local address 127.5.198.194:46633)
0929 01:07:19.156881 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.156886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.157243 (+ 357us) server_negotiation.cc:407] Connection header received
0929 01:07:19.157452 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.157458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.157526 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.157591 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:19.158173 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.158942 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.160996 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.162210 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.162393 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.162449 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.162774 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.162898 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260929 01:07:19.163236 8166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.156214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:46633 (local address 127.5.198.195:48501)
0929 01:07:19.157130 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:19.157147 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:19.157271 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:19.157754 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:19.157757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:19.157775 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:19.158022 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.158028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.159072 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.159078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.160850 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.160860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.162461 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.162469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.162627 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.162650 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 01:07:19.162799 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 01:07:19.162922 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":561,"threads_started":1}
I20260929 01:07:19.163586 7815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a76e04bd895451495d2a1cae3af83f0" candidate_uuid: "b7b4408c7a2b42b7bd71232da487152a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761f5e9739294980920bcdba74d00b9f" is_pre_election: true
I20260929 01:07:19.164158 7815 raft_consensus.cc:2454] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7b4408c7a2b42b7bd71232da487152a in term 0.
I20260929 01:07:19.164151 7949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a76e04bd895451495d2a1cae3af83f0" candidate_uuid: "b7b4408c7a2b42b7bd71232da487152a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" is_pre_election: true
I20260929 01:07:19.164664 7949 raft_consensus.cc:2454] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7b4408c7a2b42b7bd71232da487152a in term 0.
I20260929 01:07:19.164860 8021 leader_election.cc:302] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 761f5e9739294980920bcdba74d00b9f, b7b4408c7a2b42b7bd71232da487152a; no voters:
I20260929 01:07:19.165235 8165 raft_consensus.cc:2781] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:07:19.165372 8165 raft_consensus.cc:482] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:07:19.165503 8165 raft_consensus.cc:3037] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:19.167586 8165 raft_consensus.cc:504] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } }
I20260929 01:07:19.168108 8165 leader_election.cc:288] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [CANDIDATE]: Term 1 election: Requested vote from peers 761f5e9739294980920bcdba74d00b9f (127.5.198.193:35493), 3e85bba98ca04bb3ae2e956a44aa4aeb (127.5.198.194:46633)
I20260929 01:07:19.168643 7815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a76e04bd895451495d2a1cae3af83f0" candidate_uuid: "b7b4408c7a2b42b7bd71232da487152a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761f5e9739294980920bcdba74d00b9f"
I20260929 01:07:19.168663 7949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a76e04bd895451495d2a1cae3af83f0" candidate_uuid: "b7b4408c7a2b42b7bd71232da487152a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb"
I20260929 01:07:19.168838 7815 raft_consensus.cc:3037] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:19.168843 7949 raft_consensus.cc:3037] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:19.170890 7815 raft_consensus.cc:2454] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7b4408c7a2b42b7bd71232da487152a in term 1.
I20260929 01:07:19.170905 7949 raft_consensus.cc:2454] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7b4408c7a2b42b7bd71232da487152a in term 1.
I20260929 01:07:19.171253 8021 leader_election.cc:302] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 761f5e9739294980920bcdba74d00b9f, b7b4408c7a2b42b7bd71232da487152a; no voters:
I20260929 01:07:19.171586 8165 raft_consensus.cc:2781] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:19.172026 8165 raft_consensus.cc:686] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [term 1 LEADER]: Becoming Leader. State: Replica: b7b4408c7a2b42b7bd71232da487152a, State: Running, Role: LEADER
I20260929 01:07:19.172520 8165 consensus_queue.cc:237] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } }
I20260929 01:07:19.177101 7653 catalog_manager.cc:5495] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a reported cstate change: term changed from 0 to 1, leader changed from <none> to b7b4408c7a2b42b7bd71232da487152a (127.5.198.195). New cstate: current_term: 1 leader_uuid: "b7b4408c7a2b42b7bd71232da487152a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } health_report { overall_health: UNKNOWN } } }
W20260929 01:07:19.224287 7861 tablet.cc:2369] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:07:19.232060 8000 tablet.cc:2369] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 01:07:19 dist-test-slave-41cl krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644037, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
W20260929 01:07:19.242400 8139 tablet.cc:2369] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:07:19.248126 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.234890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39779 (local address 127.0.0.1:44494)
0929 01:07:19.235497 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:19.235512 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:19.235677 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:19.236096 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:19.236102 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:19.236873 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:19.237138 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.237147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.238142 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.238150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.239074 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.239084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.239793 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.239799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.239961 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.240604 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:19.240620 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:19.242849 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:19.245558 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:19.245566 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:19.245572 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:19.246020 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:19.246413 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:19.246416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:19.246419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:19.246560 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:19.247226 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:19.247232 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:19.247385 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 01:07:19.247533 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 01:07:19.247738 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":349,"threads_started":1}
I20260929 01:07:19.248149 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.234872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44494 (local address 127.5.198.254:39779)
0929 01:07:19.235075 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.235080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.235599 (+ 519us) server_negotiation.cc:407] Connection header received
0929 01:07:19.235807 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.235811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.235890 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.236054 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:19.237282 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.238021 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.239204 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.239614 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.239755 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.243037 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:19.243042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:19.243045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:19.243072 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:19.245391 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:19.246146 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:19.246150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:19.246153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:19.246305 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:19.246662 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:19.246665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:19.246667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:19.247005 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:19.247148 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.247657 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.247861 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":85}
I20260929 01:07:19.252940 7653 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:44494:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 01:07:19.253818 7653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:07:19.260399 7790 tablet_service.cc:1467] Processing CreateTablet for tablet 7263728f7c114e1894a6256da2b5c482 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3bdb8aa8082a4404ac02023af187606c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:07:19.260630 7929 tablet_service.cc:1467] Processing CreateTablet for tablet 7263728f7c114e1894a6256da2b5c482 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3bdb8aa8082a4404ac02023af187606c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:07:19.261343 7790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7263728f7c114e1894a6256da2b5c482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:19.261463 7929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7263728f7c114e1894a6256da2b5c482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:19.266961 8158 tablet_bootstrap.cc:492] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f: Bootstrap starting.
I20260929 01:07:19.266510 8068 tablet_service.cc:1467] Processing CreateTablet for tablet 7263728f7c114e1894a6256da2b5c482 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3bdb8aa8082a4404ac02023af187606c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:07:19.267387 8157 tablet_bootstrap.cc:492] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Bootstrap starting.
I20260929 01:07:19.267408 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7263728f7c114e1894a6256da2b5c482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:19.268568 8158 tablet_bootstrap.cc:654] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:19.268950 8157 tablet_bootstrap.cc:654] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:19.270880 8158 tablet_bootstrap.cc:492] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f: No bootstrap required, opened a new log
I20260929 01:07:19.271055 8158 ts_tablet_manager.cc:1397] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260929 01:07:19.271688 8158 raft_consensus.cc:348] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.271911 8158 raft_consensus.cc:374] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:19.272015 8158 raft_consensus.cc:729] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761f5e9739294980920bcdba74d00b9f, State: Initialized, Role: FOLLOWER
I20260929 01:07:19.272269 8158 consensus_queue.cc:260] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.272931 8158 ts_tablet_manager.cc:1428] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 01:07:19.272750 8163 tablet_replica.cc:439] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } }
I20260929 01:07:19.273197 8163 tablet_replica.cc:442] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:07:19.277289 8159 tablet_bootstrap.cc:492] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a: Bootstrap starting.
I20260929 01:07:19.278390 8157 tablet_bootstrap.cc:492] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb: No bootstrap required, opened a new log
I20260929 01:07:19.278528 8157 ts_tablet_manager.cc:1397] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260929 01:07:19.278690 8159 tablet_bootstrap.cc:654] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:19.279176 8157 raft_consensus.cc:348] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.279341 8157 raft_consensus.cc:374] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:19.279443 8157 raft_consensus.cc:729] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e85bba98ca04bb3ae2e956a44aa4aeb, State: Initialized, Role: FOLLOWER
I20260929 01:07:19.279716 8157 consensus_queue.cc:260] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.280215 8164 tablet_replica.cc:439] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } }
I20260929 01:07:19.280602 8164 tablet_replica.cc:442] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:07:19.281085 8157 ts_tablet_manager.cc:1428] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 01:07:19.281207 8159 tablet_bootstrap.cc:492] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a: No bootstrap required, opened a new log
I20260929 01:07:19.281378 8159 ts_tablet_manager.cc:1397] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 01:07:19.282294 8159 raft_consensus.cc:348] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.282506 8159 raft_consensus.cc:374] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:19.282610 8159 raft_consensus.cc:729] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b4408c7a2b42b7bd71232da487152a, State: Initialized, Role: FOLLOWER
I20260929 01:07:19.282855 8159 consensus_queue.cc:260] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.283336 8165 tablet_replica.cc:439] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } }
I20260929 01:07:19.283572 8159 ts_tablet_manager.cc:1428] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 01:07:19.283646 8165 tablet_replica.cc:442] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:07:19.313380 8164 raft_consensus.cc:482] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:07:19.313639 8164 raft_consensus.cc:504] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.315452 8164 leader_election.cc:288] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 761f5e9739294980920bcdba74d00b9f (127.5.198.193:35493), b7b4408c7a2b42b7bd71232da487152a (127.5.198.195:42389)
I20260929 01:07:19.322391 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.315749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:49431 (local address 127.5.198.193:35493)
0929 01:07:19.315953 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.315959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.316891 (+ 932us) server_negotiation.cc:407] Connection header received
0929 01:07:19.317089 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.317093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.317149 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.317244 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:19.317818 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.318565 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.320389 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.321478 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.321640 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.321721 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.322005 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.322120 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 01:07:19.322407 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.316120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:37813 (local address 127.5.198.195:42389)
0929 01:07:19.316372 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.316380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.317037 (+ 657us) server_negotiation.cc:407] Connection header received
0929 01:07:19.317189 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.317194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.317258 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.317350 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:19.317997 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.318689 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.320384 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.321546 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.321729 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.321797 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.322045 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.322163 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:07:19.322544 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.315923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:42389 (local address 127.5.198.194:37813)
0929 01:07:19.316890 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:19.316909 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:19.317054 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:19.317370 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:19.317378 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:19.317411 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:19.317848 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.317856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.318831 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.318836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.320235 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.320246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.321751 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.321757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.321900 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.321921 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:07:19.322024 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 01:07:19.322150 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":33,"thread_start_us":413,"threads_started":1}
I20260929 01:07:19.322935 8179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.315865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:35493 (local address 127.5.198.194:49431)
0929 01:07:19.316769 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:19.316789 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:19.316933 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:19.317255 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:19.317259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:19.317280 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:19.317660 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.317669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.318688 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.318696 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.320240 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.320251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.321694 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.321700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.321843 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.321865 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:07:19.321968 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 01:07:19.322091 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":61,"spinlock_wait_cycles":23936,"thread_start_us":722,"threads_started":1}
I20260929 01:07:19.323472 7815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7263728f7c114e1894a6256da2b5c482" candidate_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761f5e9739294980920bcdba74d00b9f" is_pre_election: true
I20260929 01:07:19.323397 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7263728f7c114e1894a6256da2b5c482" candidate_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b4408c7a2b42b7bd71232da487152a" is_pre_election: true
I20260929 01:07:19.323698 7815 raft_consensus.cc:2454] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e85bba98ca04bb3ae2e956a44aa4aeb in term 0.
I20260929 01:07:19.323763 8088 raft_consensus.cc:2454] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e85bba98ca04bb3ae2e956a44aa4aeb in term 0.
I20260929 01:07:19.324167 7882 leader_election.cc:302] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e85bba98ca04bb3ae2e956a44aa4aeb, 761f5e9739294980920bcdba74d00b9f; no voters:
I20260929 01:07:19.324564 8164 raft_consensus.cc:2781] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:07:19.324685 8164 raft_consensus.cc:482] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:07:19.324782 8164 raft_consensus.cc:3037] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:19.326853 8164 raft_consensus.cc:504] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.327374 8164 leader_election.cc:288] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [CANDIDATE]: Term 1 election: Requested vote from peers 761f5e9739294980920bcdba74d00b9f (127.5.198.193:35493), b7b4408c7a2b42b7bd71232da487152a (127.5.198.195:42389)
I20260929 01:07:19.327756 7815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7263728f7c114e1894a6256da2b5c482" candidate_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761f5e9739294980920bcdba74d00b9f"
I20260929 01:07:19.327838 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7263728f7c114e1894a6256da2b5c482" candidate_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b4408c7a2b42b7bd71232da487152a"
I20260929 01:07:19.328011 7815 raft_consensus.cc:3037] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:19.328049 8088 raft_consensus.cc:3037] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:19.330255 7815 raft_consensus.cc:2454] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e85bba98ca04bb3ae2e956a44aa4aeb in term 1.
I20260929 01:07:19.330319 8088 raft_consensus.cc:2454] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e85bba98ca04bb3ae2e956a44aa4aeb in term 1.
I20260929 01:07:19.330593 7882 leader_election.cc:302] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e85bba98ca04bb3ae2e956a44aa4aeb, 761f5e9739294980920bcdba74d00b9f; no voters:
I20260929 01:07:19.330854 8164 raft_consensus.cc:2781] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:19.331283 8164 raft_consensus.cc:686] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 LEADER]: Becoming Leader. State: Replica: 3e85bba98ca04bb3ae2e956a44aa4aeb, State: Running, Role: LEADER
I20260929 01:07:19.331748 8164 consensus_queue.cc:237] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.335671 8180 tablet_replica.cc:439] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb: TxnStatusTablet state changed. Reason: New leader 3e85bba98ca04bb3ae2e956a44aa4aeb. Latest consensus state: current_term: 1 leader_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } }
I20260929 01:07:19.335937 8180 tablet_replica.cc:442] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:07:19.336261 7652 catalog_manager.cc:5495] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e85bba98ca04bb3ae2e956a44aa4aeb (127.5.198.194). New cstate: current_term: 1 leader_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } health_report { overall_health: UNKNOWN } } }
I20260929 01:07:19.336643 8182 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 01:07:19.336892 8182 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 01:07:19.363925 8156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.357830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44376 (local address 127.5.198.194:46633)
0929 01:07:19.358030 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.358036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.358273 (+ 237us) server_negotiation.cc:407] Connection header received
0929 01:07:19.358556 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.358560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.358653 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.358791 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:07:19.359135 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.359949 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.361431 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.361842 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.361985 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.362529 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:07:19.363120 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:07:19.363297 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.363534 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.363615 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 01:07:19.363952 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.357814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:46633 (local address 127.0.0.1:44376)
0929 01:07:19.358184 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:19.358198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:19.358354 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:19.358810 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:19.358814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:19.358825 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:07:19.359009 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.359017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.360076 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.360084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.361286 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.361297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.362016 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.362022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.362261 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.362336 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:07:19.363352 (+ 1016us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:07:19.363402 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 01:07:19.363484 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 01:07:19.363627 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247}
I20260929 01:07:19.371320 8088 raft_consensus.cc:1261] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a [term 1 FOLLOWER]: Refusing update from remote peer 3e85bba98ca04bb3ae2e956a44aa4aeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:19.371342 7815 raft_consensus.cc:1261] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Refusing update from remote peer 3e85bba98ca04bb3ae2e956a44aa4aeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:19.372215 8164 consensus_queue.cc:1035] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:19.372910 8164 consensus_queue.cc:1035] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:19.386840 8163 tablet_replica.cc:439] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f: TxnStatusTablet state changed. Reason: New leader 3e85bba98ca04bb3ae2e956a44aa4aeb. Latest consensus state: current_term: 1 leader_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } }
I20260929 01:07:19.387219 8163 tablet_replica.cc:442] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:07:19.388796 8165 tablet_replica.cc:439] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a: TxnStatusTablet state changed. Reason: New leader 3e85bba98ca04bb3ae2e956a44aa4aeb. Latest consensus state: current_term: 1 leader_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } }
I20260929 01:07:19.389108 8165 tablet_replica.cc:442] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:07:19.391453 8185 mvcc.cc:205] Tried to move back new op lower bound from 7334477985255645184 to 7334477985106710528. Current Snapshot: MvccSnapshot[applied={T|T < 7334477985255645184}]
I20260929 01:07:19.402194 8165 tablet_replica.cc:439] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } }
I20260929 01:07:19.402484 8165 tablet_replica.cc:442] T 7263728f7c114e1894a6256da2b5c482 P b7b4408c7a2b42b7bd71232da487152a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:07:19.403079 8190 tablet_replica.cc:439] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } }
I20260929 01:07:19.403638 8190 tablet_replica.cc:442] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:07:19.403563 8180 tablet_replica.cc:439] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } }
I20260929 01:07:19.403883 8180 tablet_replica.cc:442] T 7263728f7c114e1894a6256da2b5c482 P 3e85bba98ca04bb3ae2e956a44aa4aeb: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:07:19.409313 8163 tablet_replica.cc:439] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } } }
I20260929 01:07:19.409569 8163 tablet_replica.cc:442] T 7263728f7c114e1894a6256da2b5c482 P 761f5e9739294980920bcdba74d00b9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:07:19.419263 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.413466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55010 (local address 127.5.198.195:42389)
0929 01:07:19.413631 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.413639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.413862 (+ 223us) server_negotiation.cc:407] Connection header received
0929 01:07:19.414128 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.414131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.414199 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.414346 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:07:19.414692 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.415733 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.417197 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.417582 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.417707 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.418137 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:07:19.418658 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:07:19.418817 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.418868 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.418943 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:07:19.434101 8106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.429130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:46633 (local address 127.5.198.195:44433)
0929 01:07:19.429338 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:19.429364 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:19.429513 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:19.429851 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:19.429856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:19.429867 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:07:19.430070 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.430078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.431098 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.431105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.432105 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.432117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.432931 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.432937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.433082 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.433131 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:07:19.433552 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:07:19.433565 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 01:07:19.433666 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 01:07:19.433806 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 01:07:19.434082 8156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.429139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:44433 (local address 127.5.198.194:46633)
0929 01:07:19.429334 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.429340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.429430 (+ 90us) server_negotiation.cc:407] Connection header received
0929 01:07:19.429667 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.429672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.429743 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.429839 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:07:19.430199 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.430969 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.432246 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.432756 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.432911 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.433361 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:07:19.433456 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:07:19.433551 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.433685 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.433760 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 01:07:19.443715 7815 raft_consensus.cc:1261] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Refusing update from remote peer b7b4408c7a2b42b7bd71232da487152a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:19.443825 7949 raft_consensus.cc:1261] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Refusing update from remote peer b7b4408c7a2b42b7bd71232da487152a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:19.444341 8167 consensus_queue.cc:1035] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [LEADER]: Connected to new peer: Peer: permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:19.444768 8165 consensus_queue.cc:1035] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:19.451061 8186 mvcc.cc:205] Tried to move back new op lower bound from 7334477985555382272 to 7334477984453926912. Current Snapshot: MvccSnapshot[applied={T|T < 7334477985555382272}]
I20260929 01:07:19.489285 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.475701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:35049 (local address 127.5.198.195:42389)
0929 01:07:19.475878 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.475884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.482176 (+ 6292us) server_negotiation.cc:407] Connection header received
0929 01:07:19.482462 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.482468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.482571 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.482635 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:07:19.483436 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.484554 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.486792 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.487443 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.487534 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.488520 (+ 986us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:07:19.488651 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:07:19.488713 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.488927 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.489006 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 01:07:19.490017 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.475779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:42389 (local address 127.5.198.194:35049)
0929 01:07:19.482068 (+ 6289us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:19.482094 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:19.482289 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:19.482865 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:19.482870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:19.482888 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:07:19.483244 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.483256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.484721 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.484726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.486606 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.486621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.488044 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.488050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.488222 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.488284 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:07:19.488794 (+ 510us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:07:19.488803 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 01:07:19.489460 (+ 657us) client_negotiation.cc:240] Negotiation successful
0929 01:07:19.489602 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6102,"thread_start_us":5884,"threads_started":1}
I20260929 01:07:19.527753 7651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:07:19.528075 7651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:07:19.531809 7651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6: Sending DeleteTablet for 3 replicas of tablet 9a76e04bd895451495d2a1cae3af83f0
I20260929 01:07:19.532835 7790 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a76e04bd895451495d2a1cae3af83f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:19 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:50904
I20260929 01:07:19.533743 8205 tablet_replica.cc:330] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f: stopping tablet replica
I20260929 01:07:19.534159 8205 raft_consensus.cc:2229] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:07:19.534538 8205 raft_consensus.cc:2258] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:19.534861 7651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:07:19.535369 7651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:07:19.537184 7929 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a76e04bd895451495d2a1cae3af83f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:19 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:44362
I20260929 01:07:19.538062 8068 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a76e04bd895451495d2a1cae3af83f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:19 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:55006
I20260929 01:07:19.539830 8205 ts_tablet_manager.cc:1905] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:19.544871 8067 tablet_service.cc:1467] Processing CreateTablet for tablet d77d11550d744203ac61efdbe352506a (DEFAULT_TABLE table=test-table [id=8ea0bf3751b345f58b6ba48740c7f1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:19.545395 8067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d77d11550d744203ac61efdbe352506a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:19.545306 7928 tablet_service.cc:1467] Processing CreateTablet for tablet d77d11550d744203ac61efdbe352506a (DEFAULT_TABLE table=test-table [id=8ea0bf3751b345f58b6ba48740c7f1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:19.545908 7928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d77d11550d744203ac61efdbe352506a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:19.549299 7790 tablet_service.cc:1467] Processing CreateTablet for tablet d77d11550d744203ac61efdbe352506a (DEFAULT_TABLE table=test-table [id=8ea0bf3751b345f58b6ba48740c7f1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:19.549691 7790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d77d11550d744203ac61efdbe352506a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:19.550523 8205 ts_tablet_manager.cc:1918] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:07:19.550657 8206 tablet_replica.cc:330] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb: stopping tablet replica
I20260929 01:07:19.550674 8205 log.cc:1196] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/9a76e04bd895451495d2a1cae3af83f0
I20260929 01:07:19.551082 8206 raft_consensus.cc:2229] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:07:19.551257 8205 ts_tablet_manager.cc:1939] T 9a76e04bd895451495d2a1cae3af83f0 P 761f5e9739294980920bcdba74d00b9f: Deleting consensus metadata
I20260929 01:07:19.551443 8206 raft_consensus.cc:2258] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:19.553188 7639 catalog_manager.cc:4841] TS 761f5e9739294980920bcdba74d00b9f (127.5.198.193:35493): tablet 9a76e04bd895451495d2a1cae3af83f0 (table test-table [id=2d090ac7908e41339ace6f157a680cd1]) successfully deleted
I20260929 01:07:19.553998 8157 tablet_bootstrap.cc:492] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb: Bootstrap starting.
I20260929 01:07:19.555634 8157 tablet_bootstrap.cc:654] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:19.557089 8158 tablet_bootstrap.cc:492] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f: Bootstrap starting.
I20260929 01:07:19.558189 8206 ts_tablet_manager.cc:1905] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:19.558889 8158 tablet_bootstrap.cc:654] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:19.561569 8159 tablet_bootstrap.cc:492] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a: Bootstrap starting.
I20260929 01:07:19.563684 8159 tablet_bootstrap.cc:654] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:19.566231 8206 ts_tablet_manager.cc:1918] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:07:19.566385 8206 log.cc:1196] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/9a76e04bd895451495d2a1cae3af83f0
I20260929 01:07:19.566860 8206 ts_tablet_manager.cc:1939] T 9a76e04bd895451495d2a1cae3af83f0 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Deleting consensus metadata
I20260929 01:07:19.567714 8207 tablet_replica.cc:330] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a: stopping tablet replica
I20260929 01:07:19.568547 7639 catalog_manager.cc:4841] TS 3e85bba98ca04bb3ae2e956a44aa4aeb (127.5.198.194:46633): tablet 9a76e04bd895451495d2a1cae3af83f0 (table test-table [id=2d090ac7908e41339ace6f157a680cd1]) successfully deleted
I20260929 01:07:19.568229 8207 raft_consensus.cc:2229] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:07:19.570569 8207 raft_consensus.cc:2258] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:19.572114 8158 tablet_bootstrap.cc:492] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f: No bootstrap required, opened a new log
I20260929 01:07:19.572305 8158 ts_tablet_manager.cc:1397] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20260929 01:07:19.573235 8158 raft_consensus.cc:348] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.573560 8158 raft_consensus.cc:374] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:19.573632 8158 raft_consensus.cc:729] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761f5e9739294980920bcdba74d00b9f, State: Initialized, Role: FOLLOWER
I20260929 01:07:19.573976 8158 consensus_queue.cc:260] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.575503 8158 ts_tablet_manager.cc:1428] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 01:07:19.577551 8207 ts_tablet_manager.cc:1905] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:19.577620 8159 tablet_bootstrap.cc:492] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a: No bootstrap required, opened a new log
I20260929 01:07:19.577841 8159 ts_tablet_manager.cc:1397] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20260929 01:07:19.578881 8159 raft_consensus.cc:348] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.579206 8159 raft_consensus.cc:374] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:19.579312 8159 raft_consensus.cc:729] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b4408c7a2b42b7bd71232da487152a, State: Initialized, Role: FOLLOWER
I20260929 01:07:19.579587 8159 consensus_queue.cc:260] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.580153 8159 ts_tablet_manager.cc:1428] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 01:07:19.582994 8157 tablet_bootstrap.cc:492] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb: No bootstrap required, opened a new log
I20260929 01:07:19.583195 8157 ts_tablet_manager.cc:1397] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.010s
I20260929 01:07:19.584403 8157 raft_consensus.cc:348] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.584730 8157 raft_consensus.cc:374] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:19.584810 8157 raft_consensus.cc:729] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e85bba98ca04bb3ae2e956a44aa4aeb, State: Initialized, Role: FOLLOWER
I20260929 01:07:19.585673 8157 consensus_queue.cc:260] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.587252 8157 ts_tablet_manager.cc:1428] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 01:07:19.587546 8207 ts_tablet_manager.cc:1918] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:07:19.587729 8207 log.cc:1196] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/9a76e04bd895451495d2a1cae3af83f0
I20260929 01:07:19.588320 8207 ts_tablet_manager.cc:1939] T 9a76e04bd895451495d2a1cae3af83f0 P b7b4408c7a2b42b7bd71232da487152a: Deleting consensus metadata
I20260929 01:07:19.591049 7637 catalog_manager.cc:4841] TS b7b4408c7a2b42b7bd71232da487152a (127.5.198.195:42389): tablet 9a76e04bd895451495d2a1cae3af83f0 (table test-table [id=2d090ac7908e41339ace6f157a680cd1]) successfully deleted
I20260929 01:07:19.637460 8163 raft_consensus.cc:482] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:07:19.637702 8163 raft_consensus.cc:504] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.639514 8163 leader_election.cc:288] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e85bba98ca04bb3ae2e956a44aa4aeb (127.5.198.194:46633), b7b4408c7a2b42b7bd71232da487152a (127.5.198.195:42389)
I20260929 01:07:19.659416 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.640445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:55025 (local address 127.5.198.195:42389)
0929 01:07:19.640655 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.640661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.651716 (+ 11055us) server_negotiation.cc:407] Connection header received
0929 01:07:19.651986 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.651992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.652080 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.652153 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:19.652999 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.654150 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.656785 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.658169 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.658289 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.658384 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.658943 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.659089 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 01:07:19.660209 8211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.640033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:42389 (local address 127.5.198.193:55025)
0929 01:07:19.651604 (+ 11571us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:19.651626 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:19.651817 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:19.652347 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:19.652352 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:19.652391 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:19.652803 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.652815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.654330 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.654335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.656608 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.656622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.658577 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.658586 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.658750 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.658782 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 01:07:19.659671 (+ 889us) client_negotiation.cc:240] Negotiation successful
0929 01:07:19.659809 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11432,"mutex_wait_us":39,"thread_start_us":10829,"threads_started":1}
I20260929 01:07:19.660732 8156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.640121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:36763 (local address 127.5.198.194:46633)
0929 01:07:19.640309 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.640316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.651303 (+ 10987us) server_negotiation.cc:407] Connection header received
0929 01:07:19.651740 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.651746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.651826 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.651889 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:19.652704 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.653837 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.657877 (+ 4040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.659266 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.659414 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.659484 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.660257 (+ 773us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.660424 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:07:19.660828 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77d11550d744203ac61efdbe352506a" candidate_uuid: "761f5e9739294980920bcdba74d00b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b4408c7a2b42b7bd71232da487152a" is_pre_election: true
I20260929 01:07:19.661079 8088 raft_consensus.cc:2454] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 761f5e9739294980920bcdba74d00b9f in term 0.
I20260929 01:07:19.661638 7740 leader_election.cc:302] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 761f5e9739294980920bcdba74d00b9f, b7b4408c7a2b42b7bd71232da487152a; no voters:
I20260929 01:07:19.662099 8163 raft_consensus.cc:2781] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:07:19.662309 8163 raft_consensus.cc:482] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:07:19.662439 8163 raft_consensus.cc:3037] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:19.663116 8212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.639731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:46633 (local address 127.5.198.193:36763)
0929 01:07:19.651183 (+ 11452us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:19.651204 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:19.651561 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:19.652074 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:19.652079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:19.652105 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:19.652513 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.652526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.654020 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.654026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.657712 (+ 3686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.657726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.659879 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.659885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.660065 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.660100 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 01:07:19.661649 (+ 1549us) client_negotiation.cc:240] Negotiation successful
0929 01:07:19.661788 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11302,"mutex_wait_us":3,"thread_start_us":11583,"threads_started":1}
I20260929 01:07:19.664230 7949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77d11550d744203ac61efdbe352506a" candidate_uuid: "761f5e9739294980920bcdba74d00b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" is_pre_election: true
I20260929 01:07:19.664527 7949 raft_consensus.cc:2454] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 761f5e9739294980920bcdba74d00b9f in term 0.
I20260929 01:07:19.665532 8163 raft_consensus.cc:504] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.666177 8163 leader_election.cc:288] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [CANDIDATE]: Term 1 election: Requested vote from peers 3e85bba98ca04bb3ae2e956a44aa4aeb (127.5.198.194:46633), b7b4408c7a2b42b7bd71232da487152a (127.5.198.195:42389)
I20260929 01:07:19.666644 7949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77d11550d744203ac61efdbe352506a" candidate_uuid: "761f5e9739294980920bcdba74d00b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb"
I20260929 01:07:19.666757 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d77d11550d744203ac61efdbe352506a" candidate_uuid: "761f5e9739294980920bcdba74d00b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b4408c7a2b42b7bd71232da487152a"
I20260929 01:07:19.666837 7949 raft_consensus.cc:3037] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:19.666947 8088 raft_consensus.cc:3037] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:19.669955 8088 raft_consensus.cc:2454] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 761f5e9739294980920bcdba74d00b9f in term 1.
I20260929 01:07:19.670156 7949 raft_consensus.cc:2454] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 761f5e9739294980920bcdba74d00b9f in term 1.
I20260929 01:07:19.670361 7740 leader_election.cc:302] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 761f5e9739294980920bcdba74d00b9f, b7b4408c7a2b42b7bd71232da487152a; no voters:
I20260929 01:07:19.670722 8163 raft_consensus.cc:2781] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:19.671180 8163 raft_consensus.cc:686] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [term 1 LEADER]: Becoming Leader. State: Replica: 761f5e9739294980920bcdba74d00b9f, State: Running, Role: LEADER
I20260929 01:07:19.671856 8163 consensus_queue.cc:237] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:19.675594 7652 catalog_manager.cc:5495] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 761f5e9739294980920bcdba74d00b9f (127.5.198.193). New cstate: current_term: 1 leader_uuid: "761f5e9739294980920bcdba74d00b9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } health_report { overall_health: UNKNOWN } } }
I20260929 01:07:19.702977 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.696248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50906 (local address 127.5.198.193:35493)
0929 01:07:19.696427 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.696432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.696451 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:07:19.696586 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.696591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.696688 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.696849 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:07:19.697258 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.698344 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.700262 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.700942 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.701071 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.701602 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:07:19.702326 (+ 724us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:07:19.702557 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.702611 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.702690 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:07:19.721869 8156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.714565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:59499 (local address 127.5.198.194:46633)
0929 01:07:19.714930 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.714936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.715463 (+ 527us) server_negotiation.cc:407] Connection header received
0929 01:07:19.715657 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.715663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.715749 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.715834 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:07:19.716438 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.717634 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.719705 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.720424 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.720576 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.721082 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:07:19.721229 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:07:19.721331 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.721494 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.721571 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240}
I20260929 01:07:19.721988 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.714575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:46633 (local address 127.5.198.193:59499)
0929 01:07:19.715343 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:19.715363 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:19.715519 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:19.715877 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:19.715882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:19.715895 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:07:19.716280 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.716293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.717779 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.717787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.719543 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.719558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.720624 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.720634 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.720816 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.720886 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:07:19.721347 (+ 461us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:07:19.721358 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 01:07:19.721462 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 01:07:19.721595 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":415,"threads_started":1}
I20260929 01:07:19.732507 7949 raft_consensus.cc:1261] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Refusing update from remote peer 761f5e9739294980920bcdba74d00b9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:19.732790 8087 raft_consensus.cc:1261] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a [term 1 FOLLOWER]: Refusing update from remote peer 761f5e9739294980920bcdba74d00b9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:19.733235 8213 consensus_queue.cc:1035] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:19.734462 8213 consensus_queue.cc:1035] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:19.796736 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.779882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:35493 (local address 127.5.198.194:51013)
0929 01:07:19.788878 (+ 8996us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:19.788901 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:19.789035 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:19.790366 (+ 1331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:19.790375 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:19.790395 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:07:19.790632 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.790649 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.792185 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.792190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.794062 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:19.794076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.795333 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.795340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:19.795563 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.795629 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:07:19.796114 (+ 485us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:07:19.796124 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 01:07:19.796197 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 01:07:19.796361 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8811}
I20260929 01:07:19.797526 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:19.789343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:51013 (local address 127.5.198.193:35493)
0929 01:07:19.789904 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:19.789911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:19.789932 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:07:19.790005 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:19.790010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:19.790090 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:19.790183 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:07:19.790821 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.792026 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.794275 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:19.795006 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:19.795138 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:19.795827 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:07:19.795980 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:07:19.796062 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:19.797102 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0929 01:07:19.797181 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415}
W20260929 01:07:19.798631 8187 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 01:07:21.419377 7652 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:07:21.419591 7652 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:07:21.423523 7652 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6: Sending DeleteTablet for 3 replicas of tablet d77d11550d744203ac61efdbe352506a
I20260929 01:07:21.424336 7790 tablet_service.cc:1514] Processing DeleteTablet for tablet d77d11550d744203ac61efdbe352506a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:21 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:50904
I20260929 01:07:21.424424 7928 tablet_service.cc:1514] Processing DeleteTablet for tablet d77d11550d744203ac61efdbe352506a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:21 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:44362
I20260929 01:07:21.424556 8067 tablet_service.cc:1514] Processing DeleteTablet for tablet d77d11550d744203ac61efdbe352506a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:21 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:55006
I20260929 01:07:21.435606 8244 tablet_replica.cc:330] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb: stopping tablet replica
I20260929 01:07:21.435985 8244 raft_consensus.cc:2229] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:07:21.436311 8244 raft_consensus.cc:2258] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:21.436928 8245 tablet_replica.cc:330] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a: stopping tablet replica
I20260929 01:07:21.437197 8245 raft_consensus.cc:2229] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:07:21.437502 8245 raft_consensus.cc:2258] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:21.438457 8244 ts_tablet_manager.cc:1905] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:21.438882 8245 ts_tablet_manager.cc:1905] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:21.440977 8243 tablet_replica.cc:330] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f: stopping tablet replica
I20260929 01:07:21.441195 8243 raft_consensus.cc:2229] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:07:21.441663 8243 raft_consensus.cc:2258] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:21.444974 8243 ts_tablet_manager.cc:1905] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:21.446920 8245 ts_tablet_manager.cc:1918] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:07:21.446987 8244 ts_tablet_manager.cc:1918] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:07:21.447073 8245 log.cc:1196] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/d77d11550d744203ac61efdbe352506a
I20260929 01:07:21.447129 8244 log.cc:1196] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/d77d11550d744203ac61efdbe352506a
I20260929 01:07:21.447520 8245 ts_tablet_manager.cc:1939] T d77d11550d744203ac61efdbe352506a P b7b4408c7a2b42b7bd71232da487152a: Deleting consensus metadata
I20260929 01:07:21.447574 8244 ts_tablet_manager.cc:1939] T d77d11550d744203ac61efdbe352506a P 3e85bba98ca04bb3ae2e956a44aa4aeb: Deleting consensus metadata
I20260929 01:07:21.448612 7637 catalog_manager.cc:4841] TS b7b4408c7a2b42b7bd71232da487152a (127.5.198.195:42389): tablet d77d11550d744203ac61efdbe352506a (table test-table [id=8ea0bf3751b345f58b6ba48740c7f1e4]) successfully deleted
I20260929 01:07:21.448808 7639 catalog_manager.cc:4841] TS 3e85bba98ca04bb3ae2e956a44aa4aeb (127.5.198.194:46633): tablet d77d11550d744203ac61efdbe352506a (table test-table [id=8ea0bf3751b345f58b6ba48740c7f1e4]) successfully deleted
I20260929 01:07:21.450924 8243 ts_tablet_manager.cc:1918] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:07:21.451085 8243 log.cc:1196] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/d77d11550d744203ac61efdbe352506a
I20260929 01:07:21.451498 8243 ts_tablet_manager.cc:1939] T d77d11550d744203ac61efdbe352506a P 761f5e9739294980920bcdba74d00b9f: Deleting consensus metadata
I20260929 01:07:21.452409 7639 catalog_manager.cc:4841] TS 761f5e9739294980920bcdba74d00b9f (127.5.198.193:35493): tablet d77d11550d744203ac61efdbe352506a (table test-table [id=8ea0bf3751b345f58b6ba48740c7f1e4]) successfully deleted
Sep 29 01:07:21 dist-test-slave-41cl krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644041, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 01:07:21 dist-test-slave-41cl krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644041, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:21.484673 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:21.469728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.5.198.254:39779)
0929 01:07:21.470298 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:21.470303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:21.470325 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:07:21.470555 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:21.470563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:21.470667 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:21.470815 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:21.472257 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:21.473170 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:21.474514 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:21.474948 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:21.475103 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:21.479168 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:21.479174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:21.479178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:21.479211 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:21.482022 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:21.482696 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:21.482703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:21.482707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:21.482848 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:21.483290 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:21.483296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:21.483299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:21.483560 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:21.483802 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:21.484090 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 01:07:21.484295 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":327,"threads_started":1}
I20260929 01:07:21.488168 7652 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:07:21.488766 7652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:07:21.496174 7790 tablet_service.cc:1467] Processing CreateTablet for tablet 36cce491563742639b8eb2142e38268c (DEFAULT_TABLE table=test-table [id=5a2ee3aa42da4223bc11924932c2ab5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:21.496146 7928 tablet_service.cc:1467] Processing CreateTablet for tablet 36cce491563742639b8eb2142e38268c (DEFAULT_TABLE table=test-table [id=5a2ee3aa42da4223bc11924932c2ab5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:21.496672 7928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cce491563742639b8eb2142e38268c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:21.496675 7790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cce491563742639b8eb2142e38268c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:21.497496 8067 tablet_service.cc:1467] Processing CreateTablet for tablet 36cce491563742639b8eb2142e38268c (DEFAULT_TABLE table=test-table [id=5a2ee3aa42da4223bc11924932c2ab5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:21.497897 8067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cce491563742639b8eb2142e38268c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:21.504156 8255 tablet_bootstrap.cc:492] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb: Bootstrap starting.
I20260929 01:07:21.507463 8255 tablet_bootstrap.cc:654] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:21.509011 8256 tablet_bootstrap.cc:492] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f: Bootstrap starting.
I20260929 01:07:21.510604 8256 tablet_bootstrap.cc:654] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:21.512686 8254 tablet_bootstrap.cc:492] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a: Bootstrap starting.
I20260929 01:07:21.512801 8256 tablet_bootstrap.cc:492] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f: No bootstrap required, opened a new log
I20260929 01:07:21.512971 8256 ts_tablet_manager.cc:1397] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 01:07:21.513623 8256 raft_consensus.cc:348] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.513748 8255 tablet_bootstrap.cc:492] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb: No bootstrap required, opened a new log
I20260929 01:07:21.513854 8256 raft_consensus.cc:374] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:21.513895 8255 ts_tablet_manager.cc:1397] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 01:07:21.513947 8256 raft_consensus.cc:729] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761f5e9739294980920bcdba74d00b9f, State: Initialized, Role: FOLLOWER
I20260929 01:07:21.514199 8256 consensus_queue.cc:260] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.514308 8254 tablet_bootstrap.cc:654] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:21.514771 8256 ts_tablet_manager.cc:1428] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 01:07:21.514879 8255 raft_consensus.cc:348] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.515275 8255 raft_consensus.cc:374] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:21.515388 8255 raft_consensus.cc:729] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e85bba98ca04bb3ae2e956a44aa4aeb, State: Initialized, Role: FOLLOWER
I20260929 01:07:21.515996 8255 consensus_queue.cc:260] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.516654 8255 ts_tablet_manager.cc:1428] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 01:07:21.517758 8254 tablet_bootstrap.cc:492] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a: No bootstrap required, opened a new log
I20260929 01:07:21.517907 8254 ts_tablet_manager.cc:1397] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260929 01:07:21.518679 8254 raft_consensus.cc:348] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.518929 8254 raft_consensus.cc:374] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:21.519034 8254 raft_consensus.cc:729] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b4408c7a2b42b7bd71232da487152a, State: Initialized, Role: FOLLOWER
I20260929 01:07:21.519254 8254 consensus_queue.cc:260] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.520102 8254 ts_tablet_manager.cc:1428] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 01:07:21.731869 8225 raft_consensus.cc:482] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:07:21.732110 8225 raft_consensus.cc:504] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.732975 8225 leader_election.cc:288] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 761f5e9739294980920bcdba74d00b9f (127.5.198.193:35493), b7b4408c7a2b42b7bd71232da487152a (127.5.198.195:42389)
I20260929 01:07:21.733500 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cce491563742639b8eb2142e38268c" candidate_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b4408c7a2b42b7bd71232da487152a" is_pre_election: true
I20260929 01:07:21.733512 7815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cce491563742639b8eb2142e38268c" candidate_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761f5e9739294980920bcdba74d00b9f" is_pre_election: true
I20260929 01:07:21.733744 7815 raft_consensus.cc:2454] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e85bba98ca04bb3ae2e956a44aa4aeb in term 0.
I20260929 01:07:21.733767 8088 raft_consensus.cc:2454] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e85bba98ca04bb3ae2e956a44aa4aeb in term 0.
I20260929 01:07:21.734205 7882 leader_election.cc:302] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e85bba98ca04bb3ae2e956a44aa4aeb, 761f5e9739294980920bcdba74d00b9f; no voters:
I20260929 01:07:21.734508 8225 raft_consensus.cc:2781] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:07:21.734627 8225 raft_consensus.cc:482] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:07:21.734717 8225 raft_consensus.cc:3037] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:21.736729 8225 raft_consensus.cc:504] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.737287 8225 leader_election.cc:288] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [CANDIDATE]: Term 1 election: Requested vote from peers 761f5e9739294980920bcdba74d00b9f (127.5.198.193:35493), b7b4408c7a2b42b7bd71232da487152a (127.5.198.195:42389)
I20260929 01:07:21.737757 7815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cce491563742639b8eb2142e38268c" candidate_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761f5e9739294980920bcdba74d00b9f"
I20260929 01:07:21.737819 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cce491563742639b8eb2142e38268c" candidate_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b4408c7a2b42b7bd71232da487152a"
I20260929 01:07:21.737943 7815 raft_consensus.cc:3037] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:21.738101 8088 raft_consensus.cc:3037] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:21.739909 7815 raft_consensus.cc:2454] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e85bba98ca04bb3ae2e956a44aa4aeb in term 1.
I20260929 01:07:21.740061 8088 raft_consensus.cc:2454] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e85bba98ca04bb3ae2e956a44aa4aeb in term 1.
I20260929 01:07:21.740293 7882 leader_election.cc:302] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e85bba98ca04bb3ae2e956a44aa4aeb, 761f5e9739294980920bcdba74d00b9f; no voters:
I20260929 01:07:21.740585 8225 raft_consensus.cc:2781] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:21.740728 8225 raft_consensus.cc:686] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 LEADER]: Becoming Leader. State: Replica: 3e85bba98ca04bb3ae2e956a44aa4aeb, State: Running, Role: LEADER
I20260929 01:07:21.741361 8225 consensus_queue.cc:237] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.743757 7651 catalog_manager.cc:5495] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e85bba98ca04bb3ae2e956a44aa4aeb (127.5.198.194). New cstate: current_term: 1 leader_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } health_report { overall_health: UNKNOWN } } }
I20260929 01:07:21.808987 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:21.804343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.5.198.194:46633)
0929 01:07:21.804861 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:21.804865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:21.804883 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:07:21.804929 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:21.804933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:21.805017 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:21.805093 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:07:21.805472 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:21.806274 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:21.807450 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:21.807830 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:21.807919 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:21.808282 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:07:21.808382 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:07:21.808445 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:21.808584 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 01:07:21.808637 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":284,"threads_started":1}
I20260929 01:07:21.821933 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:21.816966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:59345 (local address 127.5.198.194:46633)
0929 01:07:21.817156 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:21.817167 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:21.817348 (+ 181us) server_negotiation.cc:407] Connection header received
0929 01:07:21.817525 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:21.817530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:21.817588 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:21.817666 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:07:21.818046 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:21.819096 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:21.820304 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:21.820783 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:21.820914 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:21.821240 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:07:21.821337 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:07:21.821421 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:21.821556 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 01:07:21.821624 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:07:21.822023 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:21.816748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:46633 (local address 127.5.198.194:59345)
0929 01:07:21.817263 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:21.817277 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:21.817413 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:21.817684 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:21.817688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:21.817698 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:07:21.817906 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:21.817913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:21.819219 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:21.819225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:21.820175 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:21.820185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:21.820928 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:21.820934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:21.821075 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:21.821105 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:07:21.821430 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:07:21.821441 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 01:07:21.821530 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 01:07:21.821654 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":300,"threads_started":1}
I20260929 01:07:21.829406 8088 raft_consensus.cc:1261] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a [term 1 FOLLOWER]: Refusing update from remote peer 3e85bba98ca04bb3ae2e956a44aa4aeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:21.829397 7815 raft_consensus.cc:1261] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Refusing update from remote peer 3e85bba98ca04bb3ae2e956a44aa4aeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:21.829910 8225 consensus_queue.cc:1035] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:21.830210 8180 consensus_queue.cc:1035] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:21.880409 7651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:07:21.880687 7651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:07:21.884100 7651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6: Sending DeleteTablet for 3 replicas of tablet 36cce491563742639b8eb2142e38268c
I20260929 01:07:21.884893 7790 tablet_service.cc:1514] Processing DeleteTablet for tablet 36cce491563742639b8eb2142e38268c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:21 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:50904
I20260929 01:07:21.885219 8067 tablet_service.cc:1514] Processing DeleteTablet for tablet 36cce491563742639b8eb2142e38268c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:21 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:55006
I20260929 01:07:21.885268 8243 tablet_replica.cc:330] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f: stopping tablet replica
I20260929 01:07:21.885463 8245 tablet_replica.cc:330] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a: stopping tablet replica
I20260929 01:07:21.885474 8243 raft_consensus.cc:2229] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:07:21.885653 8243 raft_consensus.cc:2258] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:21.885640 8245 raft_consensus.cc:2229] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:07:21.885840 8245 raft_consensus.cc:2258] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:21.886125 7928 tablet_service.cc:1514] Processing DeleteTablet for tablet 36cce491563742639b8eb2142e38268c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:21 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:44362
I20260929 01:07:21.886405 8244 tablet_replica.cc:330] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb: stopping tablet replica
I20260929 01:07:21.886610 8244 raft_consensus.cc:2229] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:07:21.886932 8245 ts_tablet_manager.cc:1905] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:21.887105 8244 raft_consensus.cc:2258] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:21.887440 8243 ts_tablet_manager.cc:1905] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:21.888201 8244 ts_tablet_manager.cc:1905] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:21.893116 7651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:07:21.893824 7651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:07:21.894416 8243 ts_tablet_manager.cc:1918] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:07:21.894760 8243 log.cc:1196] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/36cce491563742639b8eb2142e38268c
I20260929 01:07:21.894912 8244 ts_tablet_manager.cc:1918] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:07:21.895184 8244 log.cc:1196] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/36cce491563742639b8eb2142e38268c
I20260929 01:07:21.895502 8243 ts_tablet_manager.cc:1939] T 36cce491563742639b8eb2142e38268c P 761f5e9739294980920bcdba74d00b9f: Deleting consensus metadata
I20260929 01:07:21.895649 8244 ts_tablet_manager.cc:1939] T 36cce491563742639b8eb2142e38268c P 3e85bba98ca04bb3ae2e956a44aa4aeb: Deleting consensus metadata
I20260929 01:07:21.896720 7639 catalog_manager.cc:4841] TS 761f5e9739294980920bcdba74d00b9f (127.5.198.193:35493): tablet 36cce491563742639b8eb2142e38268c (table test-table [id=5a2ee3aa42da4223bc11924932c2ab5b]) successfully deleted
I20260929 01:07:21.897301 7639 catalog_manager.cc:4841] TS 3e85bba98ca04bb3ae2e956a44aa4aeb (127.5.198.194:46633): tablet 36cce491563742639b8eb2142e38268c (table test-table [id=5a2ee3aa42da4223bc11924932c2ab5b]) successfully deleted
I20260929 01:07:21.898509 8245 ts_tablet_manager.cc:1918] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:07:21.898654 8245 log.cc:1196] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/36cce491563742639b8eb2142e38268c
I20260929 01:07:21.899053 8245 ts_tablet_manager.cc:1939] T 36cce491563742639b8eb2142e38268c P b7b4408c7a2b42b7bd71232da487152a: Deleting consensus metadata
I20260929 01:07:21.900094 7637 catalog_manager.cc:4841] TS b7b4408c7a2b42b7bd71232da487152a (127.5.198.195:42389): tablet 36cce491563742639b8eb2142e38268c (table test-table [id=5a2ee3aa42da4223bc11924932c2ab5b]) successfully deleted
I20260929 01:07:21.901779 7790 tablet_service.cc:1467] Processing CreateTablet for tablet 85a05bc52402490898def0dc9a87c198 (DEFAULT_TABLE table=test-table [id=5e7e5d01ab424b10aeab1fc15dc06cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:21.902086 7928 tablet_service.cc:1467] Processing CreateTablet for tablet 85a05bc52402490898def0dc9a87c198 (DEFAULT_TABLE table=test-table [id=5e7e5d01ab424b10aeab1fc15dc06cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:21.902222 7790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a05bc52402490898def0dc9a87c198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:21.902218 8067 tablet_service.cc:1467] Processing CreateTablet for tablet 85a05bc52402490898def0dc9a87c198 (DEFAULT_TABLE table=test-table [id=5e7e5d01ab424b10aeab1fc15dc06cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:21.902504 7928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a05bc52402490898def0dc9a87c198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:21.902581 8067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a05bc52402490898def0dc9a87c198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:21.908603 8256 tablet_bootstrap.cc:492] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f: Bootstrap starting.
I20260929 01:07:21.909325 8255 tablet_bootstrap.cc:492] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Bootstrap starting.
I20260929 01:07:21.910609 8256 tablet_bootstrap.cc:654] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:21.910760 8255 tablet_bootstrap.cc:654] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:21.912192 8254 tablet_bootstrap.cc:492] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a: Bootstrap starting.
I20260929 01:07:21.912643 8256 tablet_bootstrap.cc:492] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f: No bootstrap required, opened a new log
I20260929 01:07:21.912848 8256 ts_tablet_manager.cc:1397] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260929 01:07:21.913834 8254 tablet_bootstrap.cc:654] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:21.913798 8256 raft_consensus.cc:348] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.914189 8256 raft_consensus.cc:374] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:21.914321 8256 raft_consensus.cc:729] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761f5e9739294980920bcdba74d00b9f, State: Initialized, Role: FOLLOWER
I20260929 01:07:21.914660 8256 consensus_queue.cc:260] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.915369 8256 ts_tablet_manager.cc:1428] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:07:21.915560 8255 tablet_bootstrap.cc:492] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb: No bootstrap required, opened a new log
I20260929 01:07:21.915726 8255 ts_tablet_manager.cc:1397] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 01:07:21.916574 8255 raft_consensus.cc:348] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.916819 8255 raft_consensus.cc:374] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:21.916908 8255 raft_consensus.cc:729] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e85bba98ca04bb3ae2e956a44aa4aeb, State: Initialized, Role: FOLLOWER
I20260929 01:07:21.917156 8255 consensus_queue.cc:260] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.917737 8255 ts_tablet_manager.cc:1428] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:07:21.919860 8254 tablet_bootstrap.cc:492] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a: No bootstrap required, opened a new log
I20260929 01:07:21.920022 8254 ts_tablet_manager.cc:1397] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260929 01:07:21.920989 8254 raft_consensus.cc:348] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.921234 8254 raft_consensus.cc:374] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:21.921309 8254 raft_consensus.cc:729] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b4408c7a2b42b7bd71232da487152a, State: Initialized, Role: FOLLOWER
I20260929 01:07:21.921586 8254 consensus_queue.cc:260] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.922686 8254 ts_tablet_manager.cc:1428] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 01:07:21.950884 8260 raft_consensus.cc:482] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:07:21.951118 8260 raft_consensus.cc:504] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.951905 8260 leader_election.cc:288] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 761f5e9739294980920bcdba74d00b9f (127.5.198.193:35493), 3e85bba98ca04bb3ae2e956a44aa4aeb (127.5.198.194:46633)
I20260929 01:07:21.952668 7949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a05bc52402490898def0dc9a87c198" candidate_uuid: "b7b4408c7a2b42b7bd71232da487152a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" is_pre_election: true
I20260929 01:07:21.952945 7949 raft_consensus.cc:2454] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7b4408c7a2b42b7bd71232da487152a in term 0.
I20260929 01:07:21.952977 7815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a05bc52402490898def0dc9a87c198" candidate_uuid: "b7b4408c7a2b42b7bd71232da487152a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761f5e9739294980920bcdba74d00b9f" is_pre_election: true
I20260929 01:07:21.953208 7815 raft_consensus.cc:2454] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7b4408c7a2b42b7bd71232da487152a in term 0.
I20260929 01:07:21.953406 8021 leader_election.cc:302] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e85bba98ca04bb3ae2e956a44aa4aeb, b7b4408c7a2b42b7bd71232da487152a; no voters:
I20260929 01:07:21.953660 8260 raft_consensus.cc:2781] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:07:21.953778 8260 raft_consensus.cc:482] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:07:21.953912 8260 raft_consensus.cc:3037] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:21.956218 8260 raft_consensus.cc:504] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.956799 8260 leader_election.cc:288] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [CANDIDATE]: Term 1 election: Requested vote from peers 761f5e9739294980920bcdba74d00b9f (127.5.198.193:35493), 3e85bba98ca04bb3ae2e956a44aa4aeb (127.5.198.194:46633)
I20260929 01:07:21.957265 7815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a05bc52402490898def0dc9a87c198" candidate_uuid: "b7b4408c7a2b42b7bd71232da487152a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761f5e9739294980920bcdba74d00b9f"
I20260929 01:07:21.957346 7949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a05bc52402490898def0dc9a87c198" candidate_uuid: "b7b4408c7a2b42b7bd71232da487152a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb"
I20260929 01:07:21.957476 7815 raft_consensus.cc:3037] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:21.957576 7949 raft_consensus.cc:3037] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:21.959604 7815 raft_consensus.cc:2454] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7b4408c7a2b42b7bd71232da487152a in term 1.
I20260929 01:07:21.960024 8021 leader_election.cc:302] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 761f5e9739294980920bcdba74d00b9f, b7b4408c7a2b42b7bd71232da487152a; no voters:
I20260929 01:07:21.960279 8260 raft_consensus.cc:2781] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:21.960650 7949 raft_consensus.cc:2454] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7b4408c7a2b42b7bd71232da487152a in term 1.
I20260929 01:07:21.961398 8260 raft_consensus.cc:686] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [term 1 LEADER]: Becoming Leader. State: Replica: b7b4408c7a2b42b7bd71232da487152a, State: Running, Role: LEADER
I20260929 01:07:21.961689 8260 consensus_queue.cc:237] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } }
I20260929 01:07:21.964622 7652 catalog_manager.cc:5495] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a reported cstate change: term changed from 0 to 1, leader changed from <none> to b7b4408c7a2b42b7bd71232da487152a (127.5.198.195). New cstate: current_term: 1 leader_uuid: "b7b4408c7a2b42b7bd71232da487152a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7b4408c7a2b42b7bd71232da487152a" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 42389 } health_report { overall_health: HEALTHY } } }
I20260929 01:07:22.010202 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:21.986899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.5.198.195:42389)
0929 01:07:22.003632 (+ 16733us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:22.003639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:22.003666 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:07:22.003734 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:22.003739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:22.003836 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:22.003975 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:07:22.004437 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:22.005646 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:22.007639 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:22.008326 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:22.008954 (+ 628us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:22.009083 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:07:22.009217 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:07:22.009702 (+ 485us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:22.009753 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 01:07:22.009838 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16595,"thread_start_us":17207,"threads_started":1}
I20260929 01:07:22.021516 7815 raft_consensus.cc:1261] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Refusing update from remote peer b7b4408c7a2b42b7bd71232da487152a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:22.021474 7949 raft_consensus.cc:1261] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Refusing update from remote peer b7b4408c7a2b42b7bd71232da487152a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:22.022051 8273 consensus_queue.cc:1035] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [LEADER]: Connected to new peer: Peer: permanent_uuid: "761f5e9739294980920bcdba74d00b9f" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:22.022914 8273 consensus_queue.cc:1035] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e85bba98ca04bb3ae2e956a44aa4aeb" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 01:07:22.056962 8226 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 01:07:23.655207 7653 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:07:23.655433 7653 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:07:23.659233 7653 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a8cbc22e7344791a2556e99ef17a9a6: Sending DeleteTablet for 3 replicas of tablet 85a05bc52402490898def0dc9a87c198
I20260929 01:07:23.660004 7790 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a05bc52402490898def0dc9a87c198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:23 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:50904
I20260929 01:07:23.660111 8067 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a05bc52402490898def0dc9a87c198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:23 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:55006
I20260929 01:07:23.660420 7928 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a05bc52402490898def0dc9a87c198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:23 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:44362
I20260929 01:07:23.662388 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 7724
I20260929 01:07:23.671518 8305 tablet_replica.cc:330] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb: stopping tablet replica
I20260929 01:07:23.671756 8304 tablet_replica.cc:330] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a: stopping tablet replica
I20260929 01:07:23.671777 8305 raft_consensus.cc:2229] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:07:23.671923 8303 tablet_replica.cc:330] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f: stopping tablet replica
I20260929 01:07:23.672036 8305 raft_consensus.cc:2258] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:23.672083 8304 raft_consensus.cc:2229] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:07:23.672137 8303 raft_consensus.cc:2229] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:07:23.672400 8303 raft_consensus.cc:2258] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:23.673053 8305 ts_tablet_manager.cc:1905] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:23.673203 8304 raft_consensus.cc:2258] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:23.673483 8303 ts_tablet_manager.cc:1905] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:23.674175 8304 ts_tablet_manager.cc:1905] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:23.680346 8304 ts_tablet_manager.cc:1918] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:07:23.680383 8303 ts_tablet_manager.cc:1918] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:07:23.680368 8305 ts_tablet_manager.cc:1918] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:07:23.680517 8303 log.cc:1196] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/85a05bc52402490898def0dc9a87c198
I20260929 01:07:23.680568 8305 log.cc:1196] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/85a05bc52402490898def0dc9a87c198
I20260929 01:07:23.680745 8304 log.cc:1196] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/85a05bc52402490898def0dc9a87c198
I20260929 01:07:23.681006 8303 ts_tablet_manager.cc:1939] T 85a05bc52402490898def0dc9a87c198 P 761f5e9739294980920bcdba74d00b9f: Deleting consensus metadata
I20260929 01:07:23.681053 8305 ts_tablet_manager.cc:1939] T 85a05bc52402490898def0dc9a87c198 P 3e85bba98ca04bb3ae2e956a44aa4aeb: Deleting consensus metadata
Sep 29 01:07:23 dist-test-slave-41cl krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644041, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:07:23.681278 8304 ts_tablet_manager.cc:1939] T 85a05bc52402490898def0dc9a87c198 P b7b4408c7a2b42b7bd71232da487152a: Deleting consensus metadata
I20260929 01:07:23.682134 7639 catalog_manager.cc:4841] TS 761f5e9739294980920bcdba74d00b9f (127.5.198.193:35493): tablet 85a05bc52402490898def0dc9a87c198 (table test-table [id=5e7e5d01ab424b10aeab1fc15dc06cd3]) successfully deleted
I20260929 01:07:23.682423 7639 catalog_manager.cc:4841] TS 3e85bba98ca04bb3ae2e956a44aa4aeb (127.5.198.194:46633): tablet 85a05bc52402490898def0dc9a87c198 (table test-table [id=5e7e5d01ab424b10aeab1fc15dc06cd3]) successfully deleted
I20260929 01:07:23.682425 7637 catalog_manager.cc:4841] TS b7b4408c7a2b42b7bd71232da487152a (127.5.198.195:42389): tablet 85a05bc52402490898def0dc9a87c198 (table test-table [id=5e7e5d01ab424b10aeab1fc15dc06cd3]) successfully deleted
I20260929 01:07:23.688259 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:23.663321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44822 (local address 127.5.198.193:35493)
0929 01:07:23.670611 (+ 7290us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:23.670619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:23.670648 (+ 29us) server_negotiation.cc:407] Connection header received
0929 01:07:23.670716 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:23.670725 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:23.670855 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:23.670986 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:23.673004 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:23.674074 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:23.675818 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:23.676479 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:23.676626 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:23.682626 (+ 6000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:23.682650 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:23.682666 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:23.682724 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:23.685643 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:23.686300 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:23.686308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:23.686311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:23.686434 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:23.686821 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:23.686825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:23.686827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:23.687152 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:23.687456 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:23.687720 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 01:07:23.687938 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7104,"thread_start_us":6981,"threads_started":1}
W20260929 01:07:23.688948 7855 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44822
W20260929 01:07:23.689535 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 7724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:07:23.689747 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 7724
I20260929 01:07:23.712334 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 7864
Sep 29 01:07:23 dist-test-slave-41cl krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644041, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:07:23.727867 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:23.713252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.5.198.194:46633)
0929 01:07:23.713764 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:23.713769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:23.713788 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:07:23.713844 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:23.713872 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:23.713968 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:23.714097 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:23.715680 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:23.716837 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:23.718122 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:23.718523 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:23.718626 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:23.722834 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:23.722847 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:23.722863 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:23.722909 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:23.725598 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:23.726196 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:23.726202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:23.726205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:23.726330 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:23.726616 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:23.726619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:23.726621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:23.726931 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:23.727210 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:23.727468 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 01:07:23.727618 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":294,"threads_started":1}
W20260929 01:07:23.728570 7994 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39686
W20260929 01:07:23.729182 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 7864: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:07:23.729308 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 7864
I20260929 01:07:23.754330 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 8003
Sep 29 01:07:23 dist-test-slave-41cl krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644041, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:07:23.769114 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:23.755033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44268 (local address 127.5.198.195:42389)
0929 01:07:23.755551 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:23.755555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:23.755571 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:07:23.755622 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:23.755626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:23.755686 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:23.755774 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:23.757177 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:23.757852 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:23.759125 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:23.759523 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:23.759612 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:23.763913 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:23.763926 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:23.763943 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:23.763979 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:23.766651 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:23.767245 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:23.767252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:23.767254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:23.767387 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:23.767728 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:23.767731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:23.767733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:23.768072 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:23.768278 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:23.768693 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 01:07:23.768863 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":262,"threads_started":1}
W20260929 01:07:23.769819 8133 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44268
W20260929 01:07:23.770467 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 8003: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:07:23.770633 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 8003
I20260929 01:07:23.792721 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 7621
I20260929 01:07:23.793987 7693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:23.911216 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 7621
2026-09-29T01:07:23Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8230 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:07:24 dist-test-slave-41cl krb5kdc[8313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:07:24 dist-test-slave-41cl krb5kdc[8313](info): set up 2 sockets
Sep 29 01:07:24 dist-test-slave-41cl krb5kdc[8313](info): commencing operation
krb5kdc: starting...
W20260929 01:07:25.013140 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:07:25 dist-test-slave-41cl krb5kdc[8313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:07:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:07:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:07:25.196319 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:36917
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:36917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:25.504675 8326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:25.504994 8326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:25.505142 8326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:25.512199 8326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:25.512337 8326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:25.512449 8326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:25.512566 8326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:25.512648 8326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:07:25.522135 8326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:36917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:36917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:25.523373 8326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:25.524737 8326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:25.536343 8334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:25.536854 8332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:25.537184 8326 server_base.cc:1034] running on GCE node
W20260929 01:07:25.537317 8331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:25.538188 8326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:25.539414 8326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:25.540634 8326 hybrid_clock.cc:648] HybridClock initialized: now 1790644045540601 us; error 41 us; skew 500 ppm
I20260929 01:07:25.541018 8326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:25 dist-test-slave-41cl krb5kdc[8313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644045, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:25.548692 8326 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:07:25.550223 8326 webserver.cc:460] Webserver started at http://127.5.198.254:40381/ using document root <none> and password file <none>
I20260929 01:07:25.550923 8326 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:25.551069 8326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:25.551376 8326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:25.554317 8326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "bd964ee76229433ab5a4956210627021"
format_stamp: "Formatted at 2026-09-29 01:07:25 on dist-test-slave-41cl"
server_key: "8049715fb6c23cb5728ed99178f336a4"
server_key_iv: "6f4261e3e2485965f46a6c4d1e54ce04"
server_key_version: "encryptionkey@0"
I20260929 01:07:25.555179 8326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "bd964ee76229433ab5a4956210627021"
format_stamp: "Formatted at 2026-09-29 01:07:25 on dist-test-slave-41cl"
server_key: "8049715fb6c23cb5728ed99178f336a4"
server_key_iv: "6f4261e3e2485965f46a6c4d1e54ce04"
server_key_version: "encryptionkey@0"
I20260929 01:07:25.560057 8326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:07:25.563485 8341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:25.564844 8326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:07:25.565026 8326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "bd964ee76229433ab5a4956210627021"
format_stamp: "Formatted at 2026-09-29 01:07:25 on dist-test-slave-41cl"
server_key: "8049715fb6c23cb5728ed99178f336a4"
server_key_iv: "6f4261e3e2485965f46a6c4d1e54ce04"
server_key_version: "encryptionkey@0"
I20260929 01:07:25.565215 8326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:25.587339 8326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:25.591934 8326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:25.592213 8326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:25.612452 8326 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:36917
I20260929 01:07:25.612525 8392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:36917 every 8 connection(s)
I20260929 01:07:25.613850 8326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:07:25.617511 8393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:25.621670 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 8326
I20260929 01:07:25.621889 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:07:25.622639 5915 external_mini_cluster.cc:1428] Setting key aa635b759ce8169f58a4f3bb52d91c8e
I20260929 01:07:25.628958 8393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021: Bootstrap starting.
I20260929 01:07:25.632966 8393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:25.634166 8393 log.cc:824] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:25.637576 8393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021: No bootstrap required, opened a new log
Sep 29 01:07:25 dist-test-slave-41cl krb5kdc[8313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:25.646662 8393 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd964ee76229433ab5a4956210627021" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 36917 } }
I20260929 01:07:25.647177 8393 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:25.647326 8393 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd964ee76229433ab5a4956210627021, State: Initialized, Role: FOLLOWER
I20260929 01:07:25.648030 8393 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd964ee76229433ab5a4956210627021" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 36917 } }
I20260929 01:07:25.648301 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:25.624679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.5.198.254:36917)
0929 01:07:25.625441 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:25.625451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:25.625492 (+ 41us) server_negotiation.cc:407] Connection header received
0929 01:07:25.627577 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:25.627599 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:25.627956 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:25.628274 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:25.630196 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:25.631522 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:25.633524 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:25.634186 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:25.634360 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:25.639509 (+ 5149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:25.639517 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:25.639533 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:25.639572 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:25.643608 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:25.644343 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:25.644350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:25.644357 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:25.644604 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:25.645035 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:25.645043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:25.645052 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:25.645650 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:25.646019 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:25.646497 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 01:07:25.646952 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":383,"threads_started":1}
I20260929 01:07:25.648303 8393 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:07:25.648582 8393 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:07:25.648815 8393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:25.651388 8393 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd964ee76229433ab5a4956210627021" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 36917 } }
I20260929 01:07:25.651988 8393 leader_election.cc:302] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd964ee76229433ab5a4956210627021; no voters:
I20260929 01:07:25.652657 8393 leader_election.cc:288] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:07:25.652804 8398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:25.653522 8398 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [term 1 LEADER]: Becoming Leader. State: Replica: bd964ee76229433ab5a4956210627021, State: Running, Role: LEADER
I20260929 01:07:25.654124 8398 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd964ee76229433ab5a4956210627021" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 36917 } }
I20260929 01:07:25.654825 8393 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:25.657851 8400 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd964ee76229433ab5a4956210627021. Latest consensus state: current_term: 1 leader_uuid: "bd964ee76229433ab5a4956210627021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd964ee76229433ab5a4956210627021" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 36917 } } }
I20260929 01:07:25.658370 8400 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:25.658296 8399 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd964ee76229433ab5a4956210627021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd964ee76229433ab5a4956210627021" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 36917 } } }
I20260929 01:07:25.658620 8399 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:25.659294 8406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:07:25.664520 8406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:07:25.673954 8406 catalog_manager.cc:1347] Generated new cluster ID: 57f5089cab9045998a70f74cfda9ccac
I20260929 01:07:25.674072 8406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:07:25.688140 8406 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:07:25.689673 8406 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:07:25.697499 8406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd964ee76229433ab5a4956210627021: Generated new TSK 0
I20260929 01:07:25.698323 8406 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:07:25.752712 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:36917
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:26.070953 8419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:26.071246 8419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:26.071403 8419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:26.078440 8419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:26.078584 8419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:26.078748 8419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:07:26.088156 8419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:36917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:26.089443 8419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:26.090694 8419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:26.103627 8424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:26.103583 8427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:26.103669 8425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:26.104866 8419 server_base.cc:1034] running on GCE node
I20260929 01:07:26.105463 8419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:26.106338 8419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:26.107553 8419 hybrid_clock.cc:648] HybridClock initialized: now 1790644046107516 us; error 72 us; skew 500 ppm
I20260929 01:07:26.107844 8419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:26 dist-test-slave-41cl krb5kdc[8313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644046, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:26.110823 8419 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:07:26.112236 8419 webserver.cc:460] Webserver started at http://127.5.198.193:34261/ using document root <none> and password file <none>
I20260929 01:07:26.112936 8419 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:26.113082 8419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:26.113377 8419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:26.116065 8419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "a11da678d27747518d23d83833a2b3fe"
format_stamp: "Formatted at 2026-09-29 01:07:26 on dist-test-slave-41cl"
server_key: "2b6399794263a7695d864a1c0d808366"
server_key_iv: "305144041773293136eb3a38bacdcedc"
server_key_version: "encryptionkey@0"
I20260929 01:07:26.116878 8419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "a11da678d27747518d23d83833a2b3fe"
format_stamp: "Formatted at 2026-09-29 01:07:26 on dist-test-slave-41cl"
server_key: "2b6399794263a7695d864a1c0d808366"
server_key_iv: "305144041773293136eb3a38bacdcedc"
server_key_version: "encryptionkey@0"
I20260929 01:07:26.121882 8419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 01:07:26.125245 8434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:26.126508 8419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:07:26.126678 8419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "a11da678d27747518d23d83833a2b3fe"
format_stamp: "Formatted at 2026-09-29 01:07:26 on dist-test-slave-41cl"
server_key: "2b6399794263a7695d864a1c0d808366"
server_key_iv: "305144041773293136eb3a38bacdcedc"
server_key_version: "encryptionkey@0"
I20260929 01:07:26.126868 8419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:26.149384 8419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:26.153364 8419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:26.153645 8419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:26.154667 8419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:26.156136 8419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:26.156234 8419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:26.156332 8419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:26.156378 8419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:26.188575 8419 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:37879
I20260929 01:07:26.188642 8546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:37879 every 8 connection(s)
I20260929 01:07:26.189813 8419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:07:26.191248 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 8419
I20260929 01:07:26.191463 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:07:26.192256 5915 external_mini_cluster.cc:1428] Setting key 0149b35368498d4377ac603627aaa94c
Sep 29 01:07:26 dist-test-slave-41cl krb5kdc[8313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644046, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
I20260929 01:07:26.213287 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:26.192665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:46947 (local address 127.5.198.254:36917)
0929 01:07:26.193132 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:26.193136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:26.194245 (+ 1109us) server_negotiation.cc:407] Connection header received
0929 01:07:26.197140 (+ 2895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:26.197146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:26.197238 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:26.197327 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:26.199377 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:26.200414 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:26.201974 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:26.202743 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:26.202884 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:26.206241 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:26.206250 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:26.206258 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:26.206301 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:26.210045 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:26.210754 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:26.210760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:26.210765 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:26.210916 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:26.211322 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:26.211327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:26.211331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:26.211698 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:26.211861 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:26.212550 (+ 689us) server_negotiation.cc:299] Negotiation successful
0929 01:07:26.212796 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":217,"threads_started":1}
I20260929 01:07:26.214005 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:26.193229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:36917 (local address 127.5.198.193:46947)
0929 01:07:26.194085 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:26.194138 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:26.196821 (+ 2683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:26.197606 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:26.197621 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:26.198392 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:26.199206 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:26.199232 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:26.200613 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:26.200617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:26.201829 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:26.201841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:26.202942 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:26.202946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:26.203167 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:26.203882 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:26.203904 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:26.206016 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:26.210200 (+ 4184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:26.210207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:26.210221 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:26.210626 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:26.211037 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:26.211040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:26.211043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:26.211207 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:26.211824 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:26.211830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:26.212145 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 01:07:26.212376 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 01:07:26.212843 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":286,"threads_started":1}
I20260929 01:07:26.215607 8547 heartbeater.cc:347] Connected to a master server at 127.5.198.254:36917
I20260929 01:07:26.215935 8547 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:26.216764 8547 heartbeater.cc:511] Master 127.5.198.254:36917 requested a full tablet report, sending...
I20260929 01:07:26.218688 8358 ts_manager.cc:194] Registered new tserver with Master: a11da678d27747518d23d83833a2b3fe (127.5.198.193:37879)
I20260929 01:07:26.220089 8358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:46947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
W20260929 01:07:27.208889 5915 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.198.194: real 1.014s user 0.004s sys 0.006s
I20260929 01:07:27.209616 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:36917
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:07:27.223016 8547 heartbeater.cc:503] Master 127.5.198.254:36917 was elected leader, sending a full tablet report...
W20260929 01:07:27.542974 8553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:27.543267 8553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:27.543460 8553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:27.550637 8553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:27.550776 8553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:27.550946 8553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:07:27.561154 8553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:36917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:27.562465 8553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:27.563779 8553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:27.576946 8558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:27.576944 8561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:27.577518 8559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:27.577621 8553 server_base.cc:1034] running on GCE node
I20260929 01:07:27.578186 8553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:27.579140 8553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:27.580363 8553 hybrid_clock.cc:648] HybridClock initialized: now 1790644047580303 us; error 50 us; skew 500 ppm
I20260929 01:07:27.580698 8553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:27 dist-test-slave-41cl krb5kdc[8313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644047, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:27.584317 8553 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:07:27.586225 8553 webserver.cc:460] Webserver started at http://127.5.198.194:43563/ using document root <none> and password file <none>
I20260929 01:07:27.587009 8553 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:27.587157 8553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:27.587518 8553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:27.590945 8553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "6be1a55f4ab541daa7b940a97861fb67"
format_stamp: "Formatted at 2026-09-29 01:07:27 on dist-test-slave-41cl"
server_key: "0f2ded0036f432e64384a034a28f06c5"
server_key_iv: "bf7e158b3aab3a4ebb55b5eeeeef77f0"
server_key_version: "encryptionkey@0"
I20260929 01:07:27.591717 8553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "6be1a55f4ab541daa7b940a97861fb67"
format_stamp: "Formatted at 2026-09-29 01:07:27 on dist-test-slave-41cl"
server_key: "0f2ded0036f432e64384a034a28f06c5"
server_key_iv: "bf7e158b3aab3a4ebb55b5eeeeef77f0"
server_key_version: "encryptionkey@0"
I20260929 01:07:27.597407 8553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 01:07:27.601182 8568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:27.602527 8553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:07:27.602710 8553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "6be1a55f4ab541daa7b940a97861fb67"
format_stamp: "Formatted at 2026-09-29 01:07:27 on dist-test-slave-41cl"
server_key: "0f2ded0036f432e64384a034a28f06c5"
server_key_iv: "bf7e158b3aab3a4ebb55b5eeeeef77f0"
server_key_version: "encryptionkey@0"
I20260929 01:07:27.602923 8553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:27.626613 8553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:27.630728 8553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:27.631095 8553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:27.632045 8553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:27.633364 8553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:27.633466 8553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:27.633613 8553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:27.633687 8553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:27.667616 8553 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:41017
I20260929 01:07:27.667692 8680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:41017 every 8 connection(s)
I20260929 01:07:27.669152 8553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:07:27.677415 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 8553
I20260929 01:07:27.677596 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:07:27.678063 5915 external_mini_cluster.cc:1428] Setting key 2507c72a1cde18cc69ae8a1e88a52cef
Sep 29 01:07:27 dist-test-slave-41cl krb5kdc[8313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644047, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:27.688853 8683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:27.671499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:55767 (local address 127.5.198.254:36917)
0929 01:07:27.671919 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:27.671925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:27.672889 (+ 964us) server_negotiation.cc:407] Connection header received
0929 01:07:27.674666 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:27.674671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:27.674768 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:27.674891 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:27.676291 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:27.677127 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:27.678808 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:27.679257 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:27.679390 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:27.682690 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:27.682695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:27.682699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:27.682727 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:27.685334 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:27.686029 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:27.686034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:27.686040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:27.686151 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:27.686565 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:27.686571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:27.686575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:27.686830 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:27.686985 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:27.688176 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0929 01:07:27.688405 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":179,"threads_started":1}
I20260929 01:07:27.689810 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:27.672025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:36917 (local address 127.5.198.194:55767)
0929 01:07:27.672730 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:27.672777 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:27.674403 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:27.675109 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:27.675117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:27.675590 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:27.676132 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:27.676149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:27.677267 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:27.677274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:27.678661 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:27.678673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:27.679469 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:27.679476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:27.679690 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:27.680364 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:27.680386 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:27.682474 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:27.685468 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:27.685477 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:27.685492 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:27.685890 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:27.686258 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:27.686264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:27.686271 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:27.686455 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:27.686957 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:27.686965 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:27.687840 (+ 875us) client_negotiation.cc:769] Sending connection context
0929 01:07:27.688104 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 01:07:27.688687 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":255,"threads_started":1}
I20260929 01:07:27.691478 8681 heartbeater.cc:347] Connected to a master server at 127.5.198.254:36917
I20260929 01:07:27.691772 8681 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:27.692571 8681 heartbeater.cc:511] Master 127.5.198.254:36917 requested a full tablet report, sending...
I20260929 01:07:27.693969 8358 ts_manager.cc:194] Registered new tserver with Master: 6be1a55f4ab541daa7b940a97861fb67 (127.5.198.194:41017)
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
I20260929 01:07:27.694736 8358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:55767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
I20260929 01:07:28.697577 8681 heartbeater.cc:503] Master 127.5.198.254:36917 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
W20260929 01:07:28.923524 5915 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.198.195: real 1.243s user 0.001s sys 0.009s
I20260929 01:07:28.924325 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:36917
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:29.249658 8687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:29.249957 8687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:29.250054 8687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:29.257248 8687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:29.257385 8687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:29.257567 8687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:07:29.267105 8687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:36917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:29.268322 8687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:29.269690 8687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:29.283002 8692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:29.283002 8695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:29.283147 8693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:29.284310 8687 server_base.cc:1034] running on GCE node
I20260929 01:07:29.284963 8687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:29.285843 8687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:29.287107 8687 hybrid_clock.cc:648] HybridClock initialized: now 1790644049287074 us; error 65 us; skew 500 ppm
I20260929 01:07:29.287446 8687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:29 dist-test-slave-41cl krb5kdc[8313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644049, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:29.290618 8687 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:07:29.292121 8687 webserver.cc:460] Webserver started at http://127.5.198.195:42151/ using document root <none> and password file <none>
I20260929 01:07:29.292897 8687 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:29.293042 8687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:29.293330 8687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:29.296335 8687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "e5f03adc36b741cca071f78995244392"
format_stamp: "Formatted at 2026-09-29 01:07:29 on dist-test-slave-41cl"
server_key: "5ecb4326cead60df7b1961496322c9e2"
server_key_iv: "3d0b87e6699dd6f66ac30ed57110fc5c"
server_key_version: "encryptionkey@0"
I20260929 01:07:29.297267 8687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "e5f03adc36b741cca071f78995244392"
format_stamp: "Formatted at 2026-09-29 01:07:29 on dist-test-slave-41cl"
server_key: "5ecb4326cead60df7b1961496322c9e2"
server_key_iv: "3d0b87e6699dd6f66ac30ed57110fc5c"
server_key_version: "encryptionkey@0"
I20260929 01:07:29.302668 8687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:07:29.306422 8702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:29.307842 8687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:07:29.308025 8687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "e5f03adc36b741cca071f78995244392"
format_stamp: "Formatted at 2026-09-29 01:07:29 on dist-test-slave-41cl"
server_key: "5ecb4326cead60df7b1961496322c9e2"
server_key_iv: "3d0b87e6699dd6f66ac30ed57110fc5c"
server_key_version: "encryptionkey@0"
I20260929 01:07:29.308225 8687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:29.323402 8687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:29.327279 8687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:29.327567 8687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:29.328542 8687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:29.329783 8687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:29.329891 8687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:29.330040 8687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:29.330116 8687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:29.363131 8687 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:40999
I20260929 01:07:29.363212 8814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:40999 every 8 connection(s)
I20260929 01:07:29.364423 8687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:07:29.372185 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 8687
I20260929 01:07:29.372400 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:07:29.372948 5915 external_mini_cluster.cc:1428] Setting key 74e1690ce4874af551334b634908e3c8
Sep 29 01:07:29 dist-test-slave-41cl krb5kdc[8313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644049, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:29.385387 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:29.367410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:47153 (local address 127.5.198.254:36917)
0929 01:07:29.367805 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:29.367809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:29.369197 (+ 1388us) server_negotiation.cc:407] Connection header received
0929 01:07:29.371033 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:29.371039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:29.371143 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:29.371258 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:29.372903 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:29.373852 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:29.375813 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:29.376353 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:29.376513 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:29.379763 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:29.379768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:29.379772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:29.379804 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:29.382700 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:29.383339 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:29.383342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:29.383345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:29.383459 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:29.383853 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:29.383856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:29.383858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:29.384091 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:29.384204 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:29.384942 (+ 738us) server_negotiation.cc:299] Negotiation successful
0929 01:07:29.385104 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":200,"threads_started":1}
I20260929 01:07:29.386267 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:29.367960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:36917 (local address 127.5.198.195:47153)
0929 01:07:29.369046 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:29.369084 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:29.370766 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:29.371510 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:29.371518 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:29.372028 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:29.372704 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:29.372725 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:29.373987 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:29.373993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:29.375666 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:29.375684 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:29.376613 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:29.376618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:29.376827 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:29.377436 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:29.377467 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:29.379576 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:29.382832 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:29.382839 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:29.382851 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:29.383226 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:29.383549 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:29.383552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:29.383554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:29.383718 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:29.384197 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:29.384203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:29.384555 (+ 352us) client_negotiation.cc:769] Sending connection context
0929 01:07:29.384768 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 01:07:29.385128 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":432,"threads_started":1}
I20260929 01:07:29.387599 8815 heartbeater.cc:347] Connected to a master server at 127.5.198.254:36917
I20260929 01:07:29.387912 8815 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:29.388767 8815 heartbeater.cc:511] Master 127.5.198.254:36917 requested a full tablet report, sending...
I20260929 01:07:29.390128 8357 ts_manager.cc:194] Registered new tserver with Master: e5f03adc36b741cca071f78995244392 (127.5.198.195:40999)
I20260929 01:07:29.390935 8357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:47153
I20260929 01:07:29.400349 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 01:07:29.429589 8817 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:07:29.427023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.5.198.254:36917)
0929 01:07:29.427252 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:29.427259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:29.427673 (+ 414us) server_negotiation.cc:407] Connection header received
0929 01:07:29.427878 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:29.427885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:29.427973 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:29.428082 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:29.429379 (+ 1297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52606: BlockingRecv error: recv got EOF from 127.0.0.1:52606 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:07:29.433383 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 8419
W20260929 01:07:29.436146 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 8419: Not authorized: Client connection negotiation failed: client connection to 127.5.198.193:37879: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:07:29.436327 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 8419
I20260929 01:07:29.452018 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 8553
W20260929 01:07:29.454821 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 8553: Not authorized: Client connection negotiation failed: client connection to 127.5.198.194:41017: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:07:29.455030 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 8553
I20260929 01:07:29.471423 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 8687
W20260929 01:07:29.474375 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 8687: Not authorized: Client connection negotiation failed: client connection to 127.5.198.195:40999: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:07:29.474483 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 8687
I20260929 01:07:29.489728 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 8326
I20260929 01:07:29.490944 8388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:29.599725 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 8326
2026-09-29T01:07:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5686 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:07:30 dist-test-slave-41cl krb5kdc[8833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:07:30 dist-test-slave-41cl krb5kdc[8833](info): set up 2 sockets
Sep 29 01:07:30 dist-test-slave-41cl krb5kdc[8833](info): commencing operation
krb5kdc: starting...
W20260929 01:07:31.916699 5915 mini_kdc.cc:121] Time spent starting KDC: real 2.275s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:07:32 dist-test-slave-41cl krb5kdc[8833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:07:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:07:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:07:32.101065 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:44925
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:37917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:44925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:32.423897 8846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:32.424192 8846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:32.424374 8846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:32.431591 8846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:32.431701 8846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:32.431851 8846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:32.431955 8846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:32.432055 8846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:07:32.441864 8846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:37917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:44925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:44925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:32.443208 8846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:32.444607 8846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:32.456712 8851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:32.456781 8852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:32.456808 8854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:32.458002 8846 server_base.cc:1034] running on GCE node
I20260929 01:07:32.458717 8846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:32.460067 8846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:32.461301 8846 hybrid_clock.cc:648] HybridClock initialized: now 1790644052461259 us; error 46 us; skew 500 ppm
I20260929 01:07:32.461639 8846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:32 dist-test-slave-41cl krb5kdc[8833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644052, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:32.464787 8846 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:07:32.466256 8846 webserver.cc:460] Webserver started at http://127.5.198.254:34879/ using document root <none> and password file <none>
I20260929 01:07:32.466918 8846 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:32.467059 8846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:32.467375 8846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:32.470309 8846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "4f7723dd1d494e6b864212829695d995"
format_stamp: "Formatted at 2026-09-29 01:07:32 on dist-test-slave-41cl"
server_key: "5de532b63790a10b97651b4787ceac0b"
server_key_iv: "7328a92693ab59b31a6fb64571439a6d"
server_key_version: "encryptionkey@0"
I20260929 01:07:32.471095 8846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "4f7723dd1d494e6b864212829695d995"
format_stamp: "Formatted at 2026-09-29 01:07:32 on dist-test-slave-41cl"
server_key: "5de532b63790a10b97651b4787ceac0b"
server_key_iv: "7328a92693ab59b31a6fb64571439a6d"
server_key_version: "encryptionkey@0"
I20260929 01:07:32.476614 8846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:07:32.480221 8861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:32.481732 8846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:07:32.481928 8846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "4f7723dd1d494e6b864212829695d995"
format_stamp: "Formatted at 2026-09-29 01:07:32 on dist-test-slave-41cl"
server_key: "5de532b63790a10b97651b4787ceac0b"
server_key_iv: "7328a92693ab59b31a6fb64571439a6d"
server_key_version: "encryptionkey@0"
I20260929 01:07:32.482124 8846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:32.508847 8846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:32.513417 8846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:32.513716 8846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:32.532088 8846 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:44925
I20260929 01:07:32.532117 8912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:44925 every 8 connection(s)
I20260929 01:07:32.533484 8846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:07:32.537367 8913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:32.537814 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 8846
I20260929 01:07:32.538015 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:07:32.538648 5915 external_mini_cluster.cc:1428] Setting key 77cf189c1dba8b21bd4f316dade48621
I20260929 01:07:32.551198 8913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995: Bootstrap starting.
Sep 29 01:07:32 dist-test-slave-41cl krb5kdc[8833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:32.555011 8913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:32.556434 8913 log.cc:824] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:32.559666 8913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995: No bootstrap required, opened a new log
I20260929 01:07:32.562409 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:32.540579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.5.198.254:44925)
0929 01:07:32.541402 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:32.541414 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:32.541452 (+ 38us) server_negotiation.cc:407] Connection header received
0929 01:07:32.543320 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:32.543341 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:32.543752 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:32.544094 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:32.545926 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:32.547213 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:32.549050 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:32.549677 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:32.549803 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:32.554200 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:32.554211 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:32.554229 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:32.554280 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:32.558005 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:32.558676 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:32.558684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:32.558692 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:32.558902 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:32.559237 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:32.559242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:32.559245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:32.559900 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:32.560223 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:32.560608 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 01:07:32.561041 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":371,"threads_started":1}
I20260929 01:07:32.566540 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:07:32.568917 8913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7723dd1d494e6b864212829695d995" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44925 } }
I20260929 01:07:32.569290 8913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:32.569397 8913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7723dd1d494e6b864212829695d995, State: Initialized, Role: FOLLOWER
I20260929 01:07:32.569941 8913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7723dd1d494e6b864212829695d995" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44925 } }
I20260929 01:07:32.570187 8913 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:07:32.570344 8913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:07:32.570513 8913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:32.572677 8913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7723dd1d494e6b864212829695d995" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44925 } }
I20260929 01:07:32.573158 8913 leader_election.cc:302] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f7723dd1d494e6b864212829695d995; no voters:
I20260929 01:07:32.573781 8913 leader_election.cc:288] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:07:32.573879 8919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:32.575218 8919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [term 1 LEADER]: Becoming Leader. State: Replica: 4f7723dd1d494e6b864212829695d995, State: Running, Role: LEADER
I20260929 01:07:32.575853 8919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7723dd1d494e6b864212829695d995" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44925 } }
I20260929 01:07:32.576148 8913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:32.578630 8921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7723dd1d494e6b864212829695d995. Latest consensus state: current_term: 1 leader_uuid: "4f7723dd1d494e6b864212829695d995" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7723dd1d494e6b864212829695d995" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44925 } } }
I20260929 01:07:32.579013 8921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:32.579576 8920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f7723dd1d494e6b864212829695d995" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7723dd1d494e6b864212829695d995" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44925 } } }
I20260929 01:07:32.579892 8920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:32.583254 8926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:07:32.589418 8926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:07:32.600569 8926 catalog_manager.cc:1347] Generated new cluster ID: d9aa81d27ef246f6a3a0e4edefd9d139
I20260929 01:07:32.600684 8926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:07:32.601254 8938 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:07:32.611545 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:32.604050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.5.198.254:44925)
0929 01:07:32.604264 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:32.604277 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:32.604858 (+ 581us) server_negotiation.cc:407] Connection header received
0929 01:07:32.605075 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:32.605081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:32.605159 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:32.605266 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:32.606218 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:32.607055 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:32.608325 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:32.608833 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:32.608969 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:32.610090 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:32.610104 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:32.610108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:32.610721 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:32.610761 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:32.610782 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:32.610900 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:32.611049 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 01:07:32.611136 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":60}
I20260929 01:07:32.640302 8926 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:07:32.642184 8926 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:07:32.652925 8926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f7723dd1d494e6b864212829695d995: Generated new TSK 0
I20260929 01:07:32.653872 8926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:07:32.662807 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 8846
I20260929 01:07:32.664017 8908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:32.755860 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 8846
2026-09-29T01:07:32Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (3149 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:07:32 dist-test-slave-41cl krb5kdc[8947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:07:32 dist-test-slave-41cl krb5kdc[8947](info): set up 2 sockets
Sep 29 01:07:32 dist-test-slave-41cl krb5kdc[8947](info): commencing operation
krb5kdc: starting...
W20260929 01:07:33.836817 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:07:33 dist-test-slave-41cl krb5kdc[8947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:07:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:07:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:07:34.020979 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:42253
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:42253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:34.331243 8960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:34.331528 8960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:34.331617 8960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:34.338727 8960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:34.338835 8960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:34.338881 8960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:34.338937 8960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:34.338981 8960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:07:34.348456 8960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:42253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:42253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:34.349768 8960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:34.351008 8960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:34.368394 8966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:34.368518 8968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:34.368481 8965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:34.369437 8960 server_base.cc:1034] running on GCE node
I20260929 01:07:34.370306 8960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:34.371680 8960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:34.372915 8960 hybrid_clock.cc:648] HybridClock initialized: now 1790644054372849 us; error 57 us; skew 500 ppm
I20260929 01:07:34.373262 8960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:34 dist-test-slave-41cl krb5kdc[8947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644054, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:34.376273 8960 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:07:34.377707 8960 webserver.cc:460] Webserver started at http://127.5.198.254:37227/ using document root <none> and password file <none>
I20260929 01:07:34.378314 8960 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:34.378405 8960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:34.378682 8960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:34.381493 8960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "c2f53fb3daaa4f4887867faed2cf21f5"
format_stamp: "Formatted at 2026-09-29 01:07:34 on dist-test-slave-41cl"
server_key: "5775855bbd1332aee23d0ce3b7729951"
server_key_iv: "10bf277275a24eeb70f9e299633168a4"
server_key_version: "encryptionkey@0"
I20260929 01:07:34.382165 8960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "c2f53fb3daaa4f4887867faed2cf21f5"
format_stamp: "Formatted at 2026-09-29 01:07:34 on dist-test-slave-41cl"
server_key: "5775855bbd1332aee23d0ce3b7729951"
server_key_iv: "10bf277275a24eeb70f9e299633168a4"
server_key_version: "encryptionkey@0"
I20260929 01:07:34.387092 8960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:07:34.390304 8975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:34.391732 8960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:07:34.391875 8960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "c2f53fb3daaa4f4887867faed2cf21f5"
format_stamp: "Formatted at 2026-09-29 01:07:34 on dist-test-slave-41cl"
server_key: "5775855bbd1332aee23d0ce3b7729951"
server_key_iv: "10bf277275a24eeb70f9e299633168a4"
server_key_version: "encryptionkey@0"
I20260929 01:07:34.392020 8960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:34.407660 8960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:34.411689 8960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:34.411960 8960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:34.430378 8960 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:42253
I20260929 01:07:34.430416 9026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:42253 every 8 connection(s)
I20260929 01:07:34.431787 8960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:07:34.435276 9027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:34.437116 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 8960
I20260929 01:07:34.437355 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:07:34.438160 5915 external_mini_cluster.cc:1428] Setting key 7d5faf7197391884c81726c99d58b37b
I20260929 01:07:34.447554 9027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5: Bootstrap starting.
I20260929 01:07:34.451382 9027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5: Neither blocks nor log segments found. Creating new log.
Sep 29 01:07:34 dist-test-slave-41cl krb5kdc[8947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:34.452553 9027 log.cc:824] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:34.455622 9027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5: No bootstrap required, opened a new log
I20260929 01:07:34.460584 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:34.440115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.5.198.254:42253)
0929 01:07:34.440912 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:34.440923 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:34.440972 (+ 49us) server_negotiation.cc:407] Connection header received
0929 01:07:34.442548 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:34.442568 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:34.442926 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:34.443244 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:34.444971 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:34.446272 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:34.447988 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:34.448685 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:34.448807 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:34.453410 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:34.453418 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:34.453432 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:34.453467 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:34.456549 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:34.457224 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:34.457242 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:34.457249 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:34.457424 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:34.457826 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:34.457838 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:34.457840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:34.458380 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:34.458702 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:34.459112 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 01:07:34.459526 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":410,"threads_started":1}
I20260929 01:07:34.464030 9027 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f53fb3daaa4f4887867faed2cf21f5" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42253 } }
I20260929 01:07:34.464376 9027 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:34.464502 9027 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f53fb3daaa4f4887867faed2cf21f5, State: Initialized, Role: FOLLOWER
I20260929 01:07:34.465078 9027 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f53fb3daaa4f4887867faed2cf21f5" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42253 } }
I20260929 01:07:34.465306 9027 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:07:34.465448 9027 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:07:34.465618 9027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:34.468250 9027 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f53fb3daaa4f4887867faed2cf21f5" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42253 } }
I20260929 01:07:34.468796 9027 leader_election.cc:302] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2f53fb3daaa4f4887867faed2cf21f5; no voters:
I20260929 01:07:34.469445 9027 leader_election.cc:288] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:07:34.469609 9032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:34.470340 9032 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [term 1 LEADER]: Becoming Leader. State: Replica: c2f53fb3daaa4f4887867faed2cf21f5, State: Running, Role: LEADER
I20260929 01:07:34.470914 9032 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f53fb3daaa4f4887867faed2cf21f5" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42253 } }
I20260929 01:07:34.471465 9027 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:34.473871 9033 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2f53fb3daaa4f4887867faed2cf21f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f53fb3daaa4f4887867faed2cf21f5" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42253 } } }
I20260929 01:07:34.474340 9033 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:34.474459 9034 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2f53fb3daaa4f4887867faed2cf21f5. Latest consensus state: current_term: 1 leader_uuid: "c2f53fb3daaa4f4887867faed2cf21f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f53fb3daaa4f4887867faed2cf21f5" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42253 } } }
I20260929 01:07:34.474860 9034 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:34.476891 9039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:07:34.482600 9039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:07:34.491557 9039 catalog_manager.cc:1347] Generated new cluster ID: 0a4354ad010744c88f00cabb33ab13a7
I20260929 01:07:34.491670 9039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:07:34.504439 9039 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:07:34.505755 9039 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:07:34.519140 9039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2f53fb3daaa4f4887867faed2cf21f5: Generated new TSK 0
I20260929 01:07:34.520044 9039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:07:34.581110 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42253
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:34.907701 9053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:34.908138 9053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:34.908346 9053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:34.915567 9053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:34.915668 9053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:34.915876 9053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:07:34.925432 9053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:34.926733 9053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:34.928125 9053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:34.941094 9058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:34.941012 9059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:34.941053 9061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:34.942067 9053 server_base.cc:1034] running on GCE node
I20260929 01:07:34.942677 9053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:34.943650 9053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:34.944913 9053 hybrid_clock.cc:648] HybridClock initialized: now 1790644054944829 us; error 39 us; skew 500 ppm
I20260929 01:07:34.945232 9053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:34 dist-test-slave-41cl krb5kdc[8947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644054, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:34.948254 9053 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:07:34.949754 9053 webserver.cc:460] Webserver started at http://127.5.198.193:37505/ using document root <none> and password file <none>
I20260929 01:07:34.950433 9053 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:34.950578 9053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:34.950883 9053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:34.953738 9053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "4b63e1636d264f35aa985a267d78463a"
format_stamp: "Formatted at 2026-09-29 01:07:34 on dist-test-slave-41cl"
server_key: "9506e73b7e5cae9d39580d358dd3550f"
server_key_iv: "7ac30a46576a60405ffb0ccd774863b5"
server_key_version: "encryptionkey@0"
I20260929 01:07:34.954540 9053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "4b63e1636d264f35aa985a267d78463a"
format_stamp: "Formatted at 2026-09-29 01:07:34 on dist-test-slave-41cl"
server_key: "9506e73b7e5cae9d39580d358dd3550f"
server_key_iv: "7ac30a46576a60405ffb0ccd774863b5"
server_key_version: "encryptionkey@0"
I20260929 01:07:34.959546 9053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 01:07:34.962976 9068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:34.964341 9053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:07:34.964543 9053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "4b63e1636d264f35aa985a267d78463a"
format_stamp: "Formatted at 2026-09-29 01:07:34 on dist-test-slave-41cl"
server_key: "9506e73b7e5cae9d39580d358dd3550f"
server_key_iv: "7ac30a46576a60405ffb0ccd774863b5"
server_key_version: "encryptionkey@0"
I20260929 01:07:34.964733 9053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:34.986721 9053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:34.990761 9053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:34.991052 9053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:34.992017 9053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:34.993382 9053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:34.993484 9053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:34.993625 9053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:34.993696 9053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:35.028779 9053 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:33511
I20260929 01:07:35.028826 9180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:33511 every 8 connection(s)
I20260929 01:07:35.030136 9053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:07:35.039814 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 9053
I20260929 01:07:35.040035 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:07:35.040745 5915 external_mini_cluster.cc:1428] Setting key bf2ccd11547684b71372271fa7f97f25
Sep 29 01:07:35 dist-test-slave-41cl krb5kdc[8947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644054, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:35.049363 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:35.032645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:48245 (local address 127.5.198.254:42253)
0929 01:07:35.033049 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:35.033054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:35.034142 (+ 1088us) server_negotiation.cc:407] Connection header received
0929 01:07:35.035910 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:35.035915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:35.036006 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:35.036123 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:35.037595 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:35.038285 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:35.039797 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:35.040286 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:35.040429 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:35.043778 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:35.043783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:35.043786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:35.043816 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:35.046451 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:35.047134 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:35.047141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:35.047151 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:35.047258 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:35.047679 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:35.047684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:35.047687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:35.047942 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:35.048091 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:35.048801 (+ 710us) server_negotiation.cc:299] Negotiation successful
0929 01:07:35.049012 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":209,"threads_started":1}
I20260929 01:07:35.050338 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:35.033204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:42253 (local address 127.5.198.193:48245)
0929 01:07:35.033974 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:35.034017 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:35.035655 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:35.036338 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:35.036345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:35.036906 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:35.037437 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:35.037454 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:35.038446 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:35.038452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:35.039648 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:35.039659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:35.040528 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:35.040536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:35.040772 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:35.041490 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:35.041512 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:35.043565 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:35.046591 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:35.046599 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:35.046614 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:35.047009 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:35.047379 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:35.047386 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:35.047390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:35.047565 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:35.048065 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:35.048072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:35.048382 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 01:07:35.048694 (+ 312us) client_negotiation.cc:240] Negotiation successful
0929 01:07:35.049189 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":268,"threads_started":1}
I20260929 01:07:35.051736 9181 heartbeater.cc:347] Connected to a master server at 127.5.198.254:42253
I20260929 01:07:35.052045 9181 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:35.052911 9181 heartbeater.cc:511] Master 127.5.198.254:42253 requested a full tablet report, sending...
I20260929 01:07:35.055117 8992 ts_manager.cc:194] Registered new tserver with Master: 4b63e1636d264f35aa985a267d78463a (127.5.198.193:33511)
I20260929 01:07:35.056648 8992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:48245
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:07:35.084408 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42253
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:35.400650 9187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:35.400911 9187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:35.401010 9187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:35.407948 9187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:35.408041 9187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:35.408145 9187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:07:35.417347 9187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:35.418505 9187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:35.419711 9187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:35.432703 9193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:35.432646 9192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:35.432993 9195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:35.433527 9187 server_base.cc:1034] running on GCE node
I20260929 01:07:35.434134 9187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:35.435093 9187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:35.436308 9187 hybrid_clock.cc:648] HybridClock initialized: now 1790644055436229 us; error 72 us; skew 500 ppm
I20260929 01:07:35.436662 9187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:35 dist-test-slave-41cl krb5kdc[8947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644055, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:35.439793 9187 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:07:35.441334 9187 webserver.cc:460] Webserver started at http://127.5.198.194:41177/ using document root <none> and password file <none>
I20260929 01:07:35.442001 9187 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:35.442140 9187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:35.442432 9187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:35.445252 9187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "4d731de634d747bea1a2fe8902de11f3"
format_stamp: "Formatted at 2026-09-29 01:07:35 on dist-test-slave-41cl"
server_key: "d047e44b37c816892d507f0901288474"
server_key_iv: "efffe0bb7279b41f883dcf115cfbf891"
server_key_version: "encryptionkey@0"
I20260929 01:07:35.446029 9187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "4d731de634d747bea1a2fe8902de11f3"
format_stamp: "Formatted at 2026-09-29 01:07:35 on dist-test-slave-41cl"
server_key: "d047e44b37c816892d507f0901288474"
server_key_iv: "efffe0bb7279b41f883dcf115cfbf891"
server_key_version: "encryptionkey@0"
I20260929 01:07:35.451004 9187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:07:35.454403 9202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:35.455737 9187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:07:35.455935 9187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "4d731de634d747bea1a2fe8902de11f3"
format_stamp: "Formatted at 2026-09-29 01:07:35 on dist-test-slave-41cl"
server_key: "d047e44b37c816892d507f0901288474"
server_key_iv: "efffe0bb7279b41f883dcf115cfbf891"
server_key_version: "encryptionkey@0"
I20260929 01:07:35.456135 9187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:35.470500 9187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:35.474644 9187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:35.474936 9187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:35.476032 9187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:35.477494 9187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:35.477581 9187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:35.477669 9187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:35.477716 9187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:35.507499 9187 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:36939
I20260929 01:07:35.507539 9314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:36939 every 8 connection(s)
I20260929 01:07:35.508790 9187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:07:35.511791 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 9187
I20260929 01:07:35.511983 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:07:35.512611 5915 external_mini_cluster.cc:1428] Setting key fa6dce611de23ca3077a55232b02ae5e
Sep 29 01:07:35 dist-test-slave-41cl krb5kdc[8947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644055, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
I20260929 01:07:35.530407 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:35.511329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:52793 (local address 127.5.198.254:42253)
0929 01:07:35.511512 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:35.511516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:35.512997 (+ 1481us) server_negotiation.cc:407] Connection header received
0929 01:07:35.515442 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:35.515448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:35.515533 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:35.515660 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:35.517261 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:35.518008 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:35.519797 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:35.520237 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:35.520384 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:35.524164 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:35.524176 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:35.524179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:35.524212 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:35.526983 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:35.527784 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:35.527789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:35.527792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:35.527923 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:35.528420 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:35.528432 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:35.528435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:35.528793 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:35.528979 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:35.529774 (+ 795us) server_negotiation.cc:299] Negotiation successful
0929 01:07:35.530012 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:07:35.531422 9317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:35.511888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:42253 (local address 127.5.198.194:52793)
0929 01:07:35.512822 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:35.512870 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:35.515152 (+ 2282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:35.515906 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:35.515914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:35.516431 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:35.517089 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:35.517108 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:35.518192 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:35.518198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:35.519641 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:35.519654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:35.520495 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:35.520503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:35.520748 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:35.521496 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:35.521520 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:35.523912 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:35.527151 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:35.527162 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:35.527180 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:35.527629 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:35.528043 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:35.528050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:35.528054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:35.528290 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:35.528955 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:35.528965 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:35.529324 (+ 359us) client_negotiation.cc:769] Sending connection context
0929 01:07:35.529665 (+ 341us) client_negotiation.cc:240] Negotiation successful
0929 01:07:35.530211 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":343,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
I20260929 01:07:35.533037 9315 heartbeater.cc:347] Connected to a master server at 127.5.198.254:42253
I20260929 01:07:35.533378 9315 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:35.534282 9315 heartbeater.cc:511] Master 127.5.198.254:42253 requested a full tablet report, sending...
I20260929 01:07:35.535861 8992 ts_manager.cc:194] Registered new tserver with Master: 4d731de634d747bea1a2fe8902de11f3 (127.5.198.194:36939)
I20260929 01:07:35.536813 8992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:52793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:07:35.556967 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42253
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:35.882841 9320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:35.883127 9320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:35.883278 9320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:35.890691 9320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:35.890801 9320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:35.891014 9320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:07:35.900913 9320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:35.902222 9320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:35.903599 9320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:35.916429 9325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:35.916572 9326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:35.916538 9328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:35.922189 9320 server_base.cc:1034] running on GCE node
I20260929 01:07:35.922894 9320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:35.923820 9320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:35.925026 9320 hybrid_clock.cc:648] HybridClock initialized: now 1790644055924976 us; error 39 us; skew 500 ppm
I20260929 01:07:35.925354 9320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:35 dist-test-slave-41cl krb5kdc[8947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644055, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:35.936828 9320 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:07:35.938491 9320 webserver.cc:460] Webserver started at http://127.5.198.195:33497/ using document root <none> and password file <none>
I20260929 01:07:35.939224 9320 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:35.939373 9320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:35.939679 9320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:35.942749 9320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "a14bff2e37c14a6b958c619c10c61112"
format_stamp: "Formatted at 2026-09-29 01:07:35 on dist-test-slave-41cl"
server_key: "6c024e2dbcadaeca8e35604b9289ba51"
server_key_iv: "a4c639a28b53bb67507bfff69939499e"
server_key_version: "encryptionkey@0"
I20260929 01:07:35.943552 9320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "a14bff2e37c14a6b958c619c10c61112"
format_stamp: "Formatted at 2026-09-29 01:07:35 on dist-test-slave-41cl"
server_key: "6c024e2dbcadaeca8e35604b9289ba51"
server_key_iv: "a4c639a28b53bb67507bfff69939499e"
server_key_version: "encryptionkey@0"
I20260929 01:07:35.948977 9320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:07:35.952404 9335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:35.953887 9320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:07:35.954078 9320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "a14bff2e37c14a6b958c619c10c61112"
format_stamp: "Formatted at 2026-09-29 01:07:35 on dist-test-slave-41cl"
server_key: "6c024e2dbcadaeca8e35604b9289ba51"
server_key_iv: "a4c639a28b53bb67507bfff69939499e"
server_key_version: "encryptionkey@0"
I20260929 01:07:35.954279 9320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:35.967839 9320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:35.971832 9320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:35.972128 9320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:35.973153 9320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:35.974416 9320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:35.974519 9320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:35.974659 9320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:35.974735 9320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:36.006752 9320 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:36451
I20260929 01:07:36.006785 9447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:36451 every 8 connection(s)
I20260929 01:07:36.008057 9320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:07:36.014700 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 9320
I20260929 01:07:36.014899 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:07:36.015321 5915 external_mini_cluster.cc:1428] Setting key 46286407968784e0a41f4a61b8a3907b
Sep 29 01:07:36 dist-test-slave-41cl krb5kdc[8947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644055, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:36.027439 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:36.010665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:46283 (local address 127.5.198.254:42253)
0929 01:07:36.010846 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:36.010850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:36.012070 (+ 1220us) server_negotiation.cc:407] Connection header received
0929 01:07:36.013964 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:36.013970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:36.014057 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:36.014165 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:36.015755 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:36.016699 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:36.018212 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:36.018610 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:36.018743 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:36.022065 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:36.022069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:36.022072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:36.022102 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:36.024663 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:36.025326 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:36.025332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:36.025336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:36.025471 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:36.025865 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:36.025871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:36.025874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:36.026117 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:36.026265 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:36.026882 (+ 617us) server_negotiation.cc:299] Negotiation successful
0929 01:07:36.027082 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 01:07:36.028432 9450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:36.011211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:42253 (local address 127.5.198.195:46283)
0929 01:07:36.011905 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:36.011960 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:36.013689 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:36.014405 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:36.014413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:36.014971 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:36.015600 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:36.015618 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:36.016832 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:36.016842 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:36.018075 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:36.018086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:36.018821 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:36.018830 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:36.019065 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:36.019734 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:36.019755 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:36.021867 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:36.024798 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:36.024807 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:36.024823 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:36.025212 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:36.025576 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:36.025582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:36.025586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:36.025760 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:36.026252 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:36.026259 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:36.026565 (+ 306us) client_negotiation.cc:769] Sending connection context
0929 01:07:36.026814 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 01:07:36.027313 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":270,"threads_started":1}
I20260929 01:07:36.029811 9448 heartbeater.cc:347] Connected to a master server at 127.5.198.254:42253
I20260929 01:07:36.030112 9448 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:36.030952 9448 heartbeater.cc:511] Master 127.5.198.254:42253 requested a full tablet report, sending...
I20260929 01:07:36.032553 8992 ts_manager.cc:194] Registered new tserver with Master: a14bff2e37c14a6b958c619c10c61112 (127.5.198.195:36451)
I20260929 01:07:36.033330 8992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:46283
I20260929 01:07:36.042415 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:07:36.059815 9181 heartbeater.cc:503] Master 127.5.198.254:42253 was elected leader, sending a full tablet report...
Sep 29 01:07:36 dist-test-slave-41cl krb5kdc[8947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644056, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 01:07:36 dist-test-slave-41cl krb5kdc[8947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644056, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:36.089978 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:36.075791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37690 (local address 127.5.198.254:42253)
0929 01:07:36.076047 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:36.076057 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:36.076515 (+ 458us) server_negotiation.cc:407] Connection header received
0929 01:07:36.076768 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:36.076775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:36.076857 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:36.076966 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:36.078388 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:36.079117 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:36.080289 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:36.080762 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:36.080861 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:36.084697 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:36.084702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:36.084706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:36.084740 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:36.087130 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:36.087885 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:36.087893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:36.087897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:36.088041 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:36.088442 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:36.088449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:36.088453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:36.088834 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:36.089087 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:36.089400 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 01:07:36.089618 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 01:07:36.090590 8992 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37690
I20260929 01:07:36.092527 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 9053
Sep 29 01:07:36 dist-test-slave-41cl krb5kdc[8947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644056, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:07:36.107399 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:36.093476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.5.198.193:33511)
0929 01:07:36.094083 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:36.094090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:36.094113 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:07:36.094169 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:36.094175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:36.094369 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:36.094510 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:36.095644 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:36.096857 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:36.098038 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:36.098572 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:36.098671 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:36.102441 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:36.102453 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:36.102467 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:36.102502 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:36.105190 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:36.105764 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:36.105770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:36.105773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:36.105882 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:36.106195 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:36.106199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:36.106201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:36.106509 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:36.106715 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:36.106914 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 01:07:36.107121 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":305,"threads_started":1}
W20260929 01:07:36.108675 9176 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51874
W20260929 01:07:36.109716 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 9053: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:07:36.109899 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 9053
I20260929 01:07:36.125327 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 9187
Sep 29 01:07:36 dist-test-slave-41cl krb5kdc[8947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644056, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:07:36.140153 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:36.126125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.5.198.194:36939)
0929 01:07:36.126683 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:36.126690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:36.126715 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:07:36.126775 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:36.126782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:36.127002 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:36.127126 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:36.128194 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:36.129236 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:36.130410 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:36.130925 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:36.131033 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:36.134919 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:36.134929 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:36.134943 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:36.134972 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:36.137670 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:36.138238 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:36.138244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:36.138247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:36.138357 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:36.138719 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:36.138724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:36.138726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:36.139070 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:36.139299 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:36.139556 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 01:07:36.139769 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":296,"threads_started":1}
W20260929 01:07:36.141443 9310 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39006
W20260929 01:07:36.142580 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 9187: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:07:36.142737 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 9187
I20260929 01:07:36.158329 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 9320
Sep 29 01:07:36 dist-test-slave-41cl krb5kdc[8947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644056, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:07:36.174275 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:36.159350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.5.198.195:36451)
0929 01:07:36.159987 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:36.159993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:36.160023 (+ 30us) server_negotiation.cc:407] Connection header received
0929 01:07:36.160084 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:36.160089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:36.160296 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:36.160429 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:36.161779 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:36.162802 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:36.163957 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:36.164593 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:36.164693 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:36.169016 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:36.169027 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:36.169041 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:36.169072 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:36.171899 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:36.172633 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:36.172645 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:36.172651 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:36.172780 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:36.173089 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:36.173092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:36.173094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:36.173406 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:36.173629 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:36.173818 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 01:07:36.174007 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":319,"threads_started":1}
W20260929 01:07:36.175483 9443 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33066
W20260929 01:07:36.176656 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 9320: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:07:36.176795 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 9320
I20260929 01:07:36.192551 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 8960
I20260929 01:07:36.193688 9022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:36.295465 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 8960
2026-09-29T01:07:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3545 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:07:36 dist-test-slave-41cl krb5kdc[9465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:07:36 dist-test-slave-41cl krb5kdc[9465](info): set up 2 sockets
Sep 29 01:07:36 dist-test-slave-41cl krb5kdc[9465](info): commencing operation
krb5kdc: starting...
W20260929 01:07:37.397754 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:07:37 dist-test-slave-41cl krb5kdc[9465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:07:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:07:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:07:37.591158 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:43631
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:43631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:37.906438 9478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:37.906757 9478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:37.906919 9478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:37.914091 9478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:37.914228 9478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:37.914337 9478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:37.914435 9478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:37.914520 9478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:07:37.924165 9478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:43631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:43631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:37.925491 9478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:37.926748 9478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:37.938112 9486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:37.938213 9483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:37.938230 9484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:37.938875 9478 server_base.cc:1034] running on GCE node
I20260929 01:07:37.939675 9478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:37.941054 9478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:37.942274 9478 hybrid_clock.cc:648] HybridClock initialized: now 1790644057942236 us; error 60 us; skew 500 ppm
I20260929 01:07:37.942605 9478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:37 dist-test-slave-41cl krb5kdc[9465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644057, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:37.945698 9478 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:07:37.947110 9478 webserver.cc:460] Webserver started at http://127.5.198.254:34879/ using document root <none> and password file <none>
I20260929 01:07:37.947762 9478 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:37.947906 9478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:37.948202 9478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:37.950992 9478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "96620da6d4194e32b4afc54e6dfac875"
format_stamp: "Formatted at 2026-09-29 01:07:37 on dist-test-slave-41cl"
server_key: "8c54538f92086364ca05e61bad51d172"
server_key_iv: "a809754633af2ceb3d4de6ad8c1b7366"
server_key_version: "encryptionkey@0"
I20260929 01:07:37.951781 9478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "96620da6d4194e32b4afc54e6dfac875"
format_stamp: "Formatted at 2026-09-29 01:07:37 on dist-test-slave-41cl"
server_key: "8c54538f92086364ca05e61bad51d172"
server_key_iv: "a809754633af2ceb3d4de6ad8c1b7366"
server_key_version: "encryptionkey@0"
I20260929 01:07:37.956614 9478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260929 01:07:37.959744 9493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:37.961207 9478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:07:37.961400 9478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "96620da6d4194e32b4afc54e6dfac875"
format_stamp: "Formatted at 2026-09-29 01:07:37 on dist-test-slave-41cl"
server_key: "8c54538f92086364ca05e61bad51d172"
server_key_iv: "a809754633af2ceb3d4de6ad8c1b7366"
server_key_version: "encryptionkey@0"
I20260929 01:07:37.961603 9478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:37.978662 9478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:37.982872 9478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:37.983158 9478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:38.001705 9478 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:43631
I20260929 01:07:38.001740 9544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:43631 every 8 connection(s)
I20260929 01:07:38.003022 9478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:07:38.006629 9545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:38.008687 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 9478
I20260929 01:07:38.008924 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:07:38.009678 5915 external_mini_cluster.cc:1428] Setting key a67e79a5b822494ee02fcc31877bfb58
I20260929 01:07:38.019009 9545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875: Bootstrap starting.
I20260929 01:07:38.022765 9545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:38.023980 9545 log.cc:824] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:07:38 dist-test-slave-41cl krb5kdc[9465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:38.027587 9545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875: No bootstrap required, opened a new log
I20260929 01:07:38.033792 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:38.011724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.5.198.254:43631)
0929 01:07:38.012581 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:38.012596 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:38.012635 (+ 39us) server_negotiation.cc:407] Connection header received
0929 01:07:38.014271 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:38.014294 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:38.014655 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:38.014983 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:38.016696 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:38.018042 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:38.019716 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:38.020409 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:38.020641 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:38.025700 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:38.025711 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:38.025729 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:38.025768 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:38.029579 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:38.030206 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:38.030216 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:38.030227 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:38.030433 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:38.030763 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:38.030768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:38.030771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:38.031424 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:38.031714 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:38.032004 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 01:07:38.032430 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":357,"threads_started":1}
I20260929 01:07:38.037592 9545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96620da6d4194e32b4afc54e6dfac875" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 43631 } }
I20260929 01:07:38.037895 9545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:38.037966 9545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96620da6d4194e32b4afc54e6dfac875, State: Initialized, Role: FOLLOWER
I20260929 01:07:38.038475 9545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96620da6d4194e32b4afc54e6dfac875" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 43631 } }
I20260929 01:07:38.038671 9545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:07:38.038761 9545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:07:38.038877 9545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:38.040992 9545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96620da6d4194e32b4afc54e6dfac875" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 43631 } }
I20260929 01:07:38.041448 9545 leader_election.cc:302] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96620da6d4194e32b4afc54e6dfac875; no voters:
I20260929 01:07:38.042039 9545 leader_election.cc:288] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:07:38.042177 9550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:38.042814 9550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [term 1 LEADER]: Becoming Leader. State: Replica: 96620da6d4194e32b4afc54e6dfac875, State: Running, Role: LEADER
I20260929 01:07:38.043365 9550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96620da6d4194e32b4afc54e6dfac875" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 43631 } }
I20260929 01:07:38.044029 9545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:38.045753 9551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96620da6d4194e32b4afc54e6dfac875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96620da6d4194e32b4afc54e6dfac875" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 43631 } } }
I20260929 01:07:38.046088 9551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:38.046949 9555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:07:38.046805 9552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96620da6d4194e32b4afc54e6dfac875. Latest consensus state: current_term: 1 leader_uuid: "96620da6d4194e32b4afc54e6dfac875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96620da6d4194e32b4afc54e6dfac875" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 43631 } } }
I20260929 01:07:38.047106 9552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:38.051718 9555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:07:38.060159 9555 catalog_manager.cc:1347] Generated new cluster ID: 57a0a43284544dea847f3f2be62948da
I20260929 01:07:38.060262 9555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:07:38.074968 9555 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:07:38.076926 9555 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:07:38.087131 9555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96620da6d4194e32b4afc54e6dfac875: Generated new TSK 0
I20260929 01:07:38.087909 9555 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:07:38.148058 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:43631
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:38.463804 9571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:38.464105 9571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:38.464274 9571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:38.471771 9571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:38.472033 9571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:38.472232 9571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:07:38.481935 9571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:43631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:38.483171 9571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:38.484447 9571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:38.498059 9577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:38.498097 9579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:38.498126 9576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:38.498847 9571 server_base.cc:1034] running on GCE node
I20260929 01:07:38.499414 9571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:38.500330 9571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:38.501559 9571 hybrid_clock.cc:648] HybridClock initialized: now 1790644058501512 us; error 64 us; skew 500 ppm
I20260929 01:07:38.501914 9571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:38 dist-test-slave-41cl krb5kdc[9465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644058, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:38.505064 9571 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:07:38.506513 9571 webserver.cc:460] Webserver started at http://127.5.198.193:43983/ using document root <none> and password file <none>
I20260929 01:07:38.507170 9571 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:38.507313 9571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:38.507612 9571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:38.510504 9571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "baae40c73abc49729c59f0e2a0816afb"
format_stamp: "Formatted at 2026-09-29 01:07:38 on dist-test-slave-41cl"
server_key: "6ee6083eede6898dda338d8ced98ee04"
server_key_iv: "019581de712bbceaf39482fd68941450"
server_key_version: "encryptionkey@0"
I20260929 01:07:38.511263 9571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "baae40c73abc49729c59f0e2a0816afb"
format_stamp: "Formatted at 2026-09-29 01:07:38 on dist-test-slave-41cl"
server_key: "6ee6083eede6898dda338d8ced98ee04"
server_key_iv: "019581de712bbceaf39482fd68941450"
server_key_version: "encryptionkey@0"
I20260929 01:07:38.516213 9571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 01:07:38.519507 9586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:38.520931 9571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:07:38.521099 9571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "baae40c73abc49729c59f0e2a0816afb"
format_stamp: "Formatted at 2026-09-29 01:07:38 on dist-test-slave-41cl"
server_key: "6ee6083eede6898dda338d8ced98ee04"
server_key_iv: "019581de712bbceaf39482fd68941450"
server_key_version: "encryptionkey@0"
I20260929 01:07:38.521296 9571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:38.538970 9571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:38.542825 9571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:38.543114 9571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:38.544096 9571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:38.545594 9571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:38.545693 9571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:38.545857 9571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:38.545928 9571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:38.578274 9571 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:39243
I20260929 01:07:38.578313 9698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:39243 every 8 connection(s)
I20260929 01:07:38.579509 9571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:07:38.587520 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 9571
I20260929 01:07:38.587919 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:07:38.588620 5915 external_mini_cluster.cc:1428] Setting key 44cc2214c7cca3a7f019a7a6c7b2c42e
Sep 29 01:07:38 dist-test-slave-41cl krb5kdc[9465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644058, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:38.601073 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:38.582059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:52443 (local address 127.5.198.254:43631)
0929 01:07:38.582498 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:38.582502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:38.583468 (+ 966us) server_negotiation.cc:407] Connection header received
0929 01:07:38.585420 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:38.585425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:38.585514 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:38.585630 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:38.587519 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:38.588406 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:38.590568 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:38.590980 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:38.591117 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:38.595048 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:38.595054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:38.595057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:38.595088 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:38.597705 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:38.598466 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:38.598472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:38.598475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:38.598603 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:38.599078 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:38.599083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:38.599087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:38.599341 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:38.599491 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:38.600440 (+ 949us) server_negotiation.cc:299] Negotiation successful
0929 01:07:38.600692 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":240,"threads_started":1}
I20260929 01:07:38.602193 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:38.582546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:43631 (local address 127.5.198.193:52443)
0929 01:07:38.583308 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:38.583353 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:38.585151 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:38.585865 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:38.585873 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:38.586510 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:38.587281 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:38.587328 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:38.588615 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:38.588620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:38.590401 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:38.590413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:38.591223 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:38.591238 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:38.591481 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:38.592167 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:38.592195 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:38.594791 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:38.597850 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:38.597860 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:38.597878 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:38.598329 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:38.598725 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:38.598735 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:38.598739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:38.598934 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:38.599473 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:38.599480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:38.599861 (+ 381us) client_negotiation.cc:769] Sending connection context
0929 01:07:38.600356 (+ 495us) client_negotiation.cc:240] Negotiation successful
0929 01:07:38.600925 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":283,"threads_started":1}
I20260929 01:07:38.604022 9699 heartbeater.cc:347] Connected to a master server at 127.5.198.254:43631
I20260929 01:07:38.604324 9699 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:38.605080 9699 heartbeater.cc:511] Master 127.5.198.254:43631 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
I20260929 01:07:38.607286 9510 ts_manager.cc:194] Registered new tserver with Master: baae40c73abc49729c59f0e2a0816afb (127.5.198.193:39243)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
I20260929 01:07:38.608914 9510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:52443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:07:38.632931 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:43631
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:38.942778 9705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:38.943049 9705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:38.943141 9705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:38.950130 9705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:38.950232 9705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:38.950340 9705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:07:38.959738 9705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:43631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:38.960997 9705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:38.962216 9705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:38.974867 9710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:38.974792 9713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:38.974908 9711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:38.975605 9705 server_base.cc:1034] running on GCE node
I20260929 01:07:38.976125 9705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:38.977175 9705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:38.978528 9705 hybrid_clock.cc:648] HybridClock initialized: now 1790644058978482 us; error 60 us; skew 500 ppm
I20260929 01:07:38.978849 9705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:38 dist-test-slave-41cl krb5kdc[9465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644058, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:38.981880 9705 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:07:38.983256 9705 webserver.cc:460] Webserver started at http://127.5.198.194:45823/ using document root <none> and password file <none>
I20260929 01:07:38.983919 9705 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:38.984059 9705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:38.984361 9705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:38.987171 9705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "6e0c81ad9e404bd1b6990d68834832d1"
format_stamp: "Formatted at 2026-09-29 01:07:38 on dist-test-slave-41cl"
server_key: "c81e995092b87c54c4b78cb25f685f06"
server_key_iv: "1c0499c14ce2497b43a9702d52347a31"
server_key_version: "encryptionkey@0"
I20260929 01:07:38.987931 9705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "6e0c81ad9e404bd1b6990d68834832d1"
format_stamp: "Formatted at 2026-09-29 01:07:38 on dist-test-slave-41cl"
server_key: "c81e995092b87c54c4b78cb25f685f06"
server_key_iv: "1c0499c14ce2497b43a9702d52347a31"
server_key_version: "encryptionkey@0"
I20260929 01:07:38.992874 9705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 01:07:38.996085 9720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:38.997351 9705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:07:38.997510 9705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "6e0c81ad9e404bd1b6990d68834832d1"
format_stamp: "Formatted at 2026-09-29 01:07:38 on dist-test-slave-41cl"
server_key: "c81e995092b87c54c4b78cb25f685f06"
server_key_iv: "1c0499c14ce2497b43a9702d52347a31"
server_key_version: "encryptionkey@0"
I20260929 01:07:38.997697 9705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:39.011030 9705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:39.015942 9705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:39.016247 9705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:39.017388 9705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:39.018911 9705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:39.019004 9705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:39.019109 9705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:39.019162 9705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:39.053014 9705 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:36267
I20260929 01:07:39.053061 9832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:36267 every 8 connection(s)
I20260929 01:07:39.054346 9705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:07:39.060112 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 9705
I20260929 01:07:39.060361 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:07:39.060828 5915 external_mini_cluster.cc:1428] Setting key e234b37ab892567eee9da6987542752c
Sep 29 01:07:39 dist-test-slave-41cl krb5kdc[9465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644058, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:39.075532 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:39.056942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:49815 (local address 127.5.198.254:43631)
0929 01:07:39.057111 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:39.057115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:39.058245 (+ 1130us) server_negotiation.cc:407] Connection header received
0929 01:07:39.060050 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:39.060055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:39.060148 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:39.060261 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:39.061978 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.062819 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.064396 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.064933 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.065068 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:39.069157 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:39.069163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:39.069166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:39.069201 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:39.072049 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:39.072977 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:39.072990 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:39.072993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:39.073119 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:39.073601 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:39.073611 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:39.073613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:39.073902 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:39.074074 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:39.074908 (+ 834us) server_negotiation.cc:299] Negotiation successful
0929 01:07:39.075149 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 01:07:39.076717 9835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:39.057383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:43631 (local address 127.5.198.194:49815)
0929 01:07:39.058074 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:39.058112 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:39.059780 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:39.060564 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:39.060572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:39.061123 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:39.061795 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:39.061821 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.062965 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.062970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:39.064253 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:39.064264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.065151 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.065158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:39.065432 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:39.066211 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:39.066235 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:39.068888 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:39.072232 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:39.072242 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:39.072263 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:39.072835 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:39.073249 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:39.073254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:39.073258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:39.073467 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:39.074053 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:39.074062 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:39.074437 (+ 375us) client_negotiation.cc:769] Sending connection context
0929 01:07:39.074818 (+ 381us) client_negotiation.cc:240] Negotiation successful
0929 01:07:39.075403 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":271,"threads_started":1}
I20260929 01:07:39.078472 9833 heartbeater.cc:347] Connected to a master server at 127.5.198.254:43631
I20260929 01:07:39.078867 9833 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
I20260929 01:07:39.079659 9833 heartbeater.cc:511] Master 127.5.198.254:43631 requested a full tablet report, sending...
I20260929 01:07:39.081264 9510 ts_manager.cc:194] Registered new tserver with Master: 6e0c81ad9e404bd1b6990d68834832d1 (127.5.198.194:36267)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
I20260929 01:07:39.082331 9510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:49815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:07:39.107539 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:43631
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:39.419457 9838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:39.419710 9838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:39.419864 9838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:39.427063 9838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:39.427163 9838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:39.427342 9838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:07:39.436561 9838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:43631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:39.437783 9838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:39.439033 9838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:39.452026 9846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:39.452057 9843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:39.451977 9844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:39.453008 9838 server_base.cc:1034] running on GCE node
I20260929 01:07:39.453594 9838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:39.454540 9838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:39.455732 9838 hybrid_clock.cc:648] HybridClock initialized: now 1790644059455687 us; error 44 us; skew 500 ppm
I20260929 01:07:39.456030 9838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:39 dist-test-slave-41cl krb5kdc[9465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644059, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:39.459061 9838 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:07:39.460393 9838 webserver.cc:460] Webserver started at http://127.5.198.195:35551/ using document root <none> and password file <none>
I20260929 01:07:39.461036 9838 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:39.461133 9838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:39.461364 9838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:39.464085 9838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "ac72adac67e24e95b182797afb903e2e"
format_stamp: "Formatted at 2026-09-29 01:07:39 on dist-test-slave-41cl"
server_key: "392754a55e2c04f948b246ff3379fe6a"
server_key_iv: "b1717a0d9ac3b6e979287d7d6312bc04"
server_key_version: "encryptionkey@0"
I20260929 01:07:39.464890 9838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "ac72adac67e24e95b182797afb903e2e"
format_stamp: "Formatted at 2026-09-29 01:07:39 on dist-test-slave-41cl"
server_key: "392754a55e2c04f948b246ff3379fe6a"
server_key_iv: "b1717a0d9ac3b6e979287d7d6312bc04"
server_key_version: "encryptionkey@0"
I20260929 01:07:39.469686 9838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:07:39.472904 9853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:39.474143 9838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:07:39.474263 9838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "ac72adac67e24e95b182797afb903e2e"
format_stamp: "Formatted at 2026-09-29 01:07:39 on dist-test-slave-41cl"
server_key: "392754a55e2c04f948b246ff3379fe6a"
server_key_iv: "b1717a0d9ac3b6e979287d7d6312bc04"
server_key_version: "encryptionkey@0"
I20260929 01:07:39.474424 9838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:39.503926 9838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:39.508164 9838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:39.508386 9838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:39.509303 9838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:39.510500 9838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:39.510567 9838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:39.510646 9838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:39.510694 9838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:39.543272 9838 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:40385
I20260929 01:07:39.543341 9965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:40385 every 8 connection(s)
I20260929 01:07:39.544672 9838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:07:39.545235 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 9838
I20260929 01:07:39.545499 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:07:39.546056 5915 external_mini_cluster.cc:1428] Setting key 130d7e8f74062ed362986cd51953d440
Sep 29 01:07:39 dist-test-slave-41cl krb5kdc[9465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644059, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:39.564381 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:39.547410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:59105 (local address 127.5.198.254:43631)
0929 01:07:39.547616 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:39.547624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:39.549033 (+ 1409us) server_negotiation.cc:407] Connection header received
0929 01:07:39.550990 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:39.550994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:39.551059 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:39.551146 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:39.552728 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.553501 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.554946 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.555325 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.555414 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:39.558717 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:39.558721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:39.558724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:39.558754 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:39.561860 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:39.562563 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:39.562567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:39.562569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:39.562678 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:39.563038 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:39.563041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:39.563043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:39.563289 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:39.563399 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:39.563992 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 01:07:39.564135 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:07:39.565448 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:39.547988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:43631 (local address 127.5.198.195:59105)
0929 01:07:39.548855 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:39.548932 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:39.550727 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:39.551404 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:39.551419 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:39.551917 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:39.552538 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:39.552565 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.553624 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.553628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:39.554831 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:39.554842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.555499 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.555504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:39.555691 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:39.556372 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:39.556395 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:39.558530 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:39.561995 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:39.562001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:39.562014 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:39.562442 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:39.562767 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:39.562770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:39.562772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:39.562941 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:39.563389 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:39.563394 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:39.563698 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 01:07:39.563910 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 01:07:39.564277 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":438,"threads_started":1}
I20260929 01:07:39.566710 9966 heartbeater.cc:347] Connected to a master server at 127.5.198.254:43631
I20260929 01:07:39.567032 9966 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:39.567751 9966 heartbeater.cc:511] Master 127.5.198.254:43631 requested a full tablet report, sending...
I20260929 01:07:39.569140 9510 ts_manager.cc:194] Registered new tserver with Master: ac72adac67e24e95b182797afb903e2e (127.5.198.195:40385)
I20260929 01:07:39.569957 9510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:59105
I20260929 01:07:39.573652 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 01:07:39 dist-test-slave-41cl krb5kdc[9465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260929 01:07:39.612622 9699 heartbeater.cc:503] Master 127.5.198.254:43631 was elected leader, sending a full tablet report...
Sep 29 01:07:39 dist-test-slave-41cl krb5kdc[9465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:07:39.622226 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:39.607517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40702 (local address 127.5.198.193:39243)
0929 01:07:39.608119 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:39.608127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:39.608153 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:07:39.608211 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:39.608218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:39.608427 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:39.608647 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:39.609946 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.610965 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.612158 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.612834 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.612996 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:39.616919 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:39.616931 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:39.616946 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:39.616984 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:39.619766 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:39.620351 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:39.620357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:39.620360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:39.620536 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:39.620967 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:39.620970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:39.620972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:39.621276 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:39.621480 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:39.621764 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 01:07:39.621949 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":276,"threads_started":1}
Sep 29 01:07:39 dist-test-slave-41cl krb5kdc[9465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:39.639364 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:39.626227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41062 (local address 127.5.198.254:43631)
0929 01:07:39.626436 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:39.626442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:39.626658 (+ 216us) server_negotiation.cc:407] Connection header received
0929 01:07:39.626759 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:39.626762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:39.626832 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:39.626908 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:39.628018 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.628769 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.630157 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.630531 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.630629 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:39.634403 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:39.634408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:39.634411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:39.634440 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:39.636959 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:39.637550 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:39.637556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:39.637560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:39.637679 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:39.638036 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:39.638043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:39.638046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:39.638338 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:39.638584 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:39.638870 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 01:07:39.639061 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 01:07:39.639915 9510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41062
I20260929 01:07:39.640959 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 9571
I20260929 01:07:39.653352 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:39.641820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40712 (local address 127.5.198.193:39243)
0929 01:07:39.642045 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:39.642052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:39.642250 (+ 198us) server_negotiation.cc:407] Connection header received
0929 01:07:39.642354 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:39.642358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:39.642450 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:39.642551 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:39.643642 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.644365 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.645870 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.646288 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.646402 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:39.648404 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:39.648409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:39.648412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:39.648441 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:39.650950 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:39.651487 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:39.651490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:39.651492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:39.651606 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:39.652021 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:39.652024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:39.652026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:39.652252 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:39.652432 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:39.652742 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 01:07:39.652995 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:07:39.654014 9694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:39.735018 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 9571
I20260929 01:07:39.751241 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 9705
Sep 29 01:07:39 dist-test-slave-41cl krb5kdc[9465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:07:39.766530 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:39.752199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33804 (local address 127.5.198.194:36267)
0929 01:07:39.752745 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:39.752753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:39.752776 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:07:39.752833 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:39.752840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:39.753053 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:39.753250 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:39.754393 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.755366 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.756701 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.757220 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.757362 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:39.761378 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:39.761387 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:39.761400 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:39.761429 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:39.763844 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:39.764498 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:39.764509 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:39.764515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:39.764653 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:39.765032 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:39.765037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:39.765040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:39.765345 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:39.765649 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:39.765943 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 01:07:39.766223 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":315,"threads_started":1}
I20260929 01:07:39.767732 9828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:39.849740 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 9705
I20260929 01:07:39.868937 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 9838
Sep 29 01:07:39 dist-test-slave-41cl krb5kdc[9465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:07:39.885468 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:39.870025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.5.198.195:40385)
0929 01:07:39.870675 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:39.870682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:39.870709 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:07:39.870768 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:39.870774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:39.870994 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:39.871170 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:39.872383 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.873478 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.874747 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:39.875307 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:39.875446 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:39.879899 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:39.879909 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:39.879952 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:39.879985 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:39.882774 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:39.883418 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:39.883427 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:39.883429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:39.883540 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:39.883897 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:39.883904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:39.883907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:39.884226 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:39.884731 (+ 505us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:39.884885 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 01:07:39.885145 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":346,"threads_started":1}
I20260929 01:07:39.886716 9961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:39.969647 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 9838
I20260929 01:07:39.985379 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 9478
I20260929 01:07:39.986567 9540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:40.092417 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 9478
2026-09-29T01:07:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3797 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:07:40 dist-test-slave-41cl krb5kdc[9985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:07:40 dist-test-slave-41cl krb5kdc[9985](info): set up 2 sockets
Sep 29 01:07:40 dist-test-slave-41cl krb5kdc[9985](info): commencing operation
krb5kdc: starting...
W20260929 01:07:41.181026 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:07:41 dist-test-slave-41cl krb5kdc[9985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:07:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:07:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:07:41.377590 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:33565
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:40661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:33565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:41.692065 9998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:41.692349 9998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:41.692521 9998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:41.699977 9998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:41.700085 9998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:41.700169 9998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:41.700277 9998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:41.700362 9998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:07:41.710078 9998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:33565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:33565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:41.711333 9998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:41.712687 9998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:41.724304 10006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:41.724342 10004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:41.724263 10003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:41.725577 9998 server_base.cc:1034] running on GCE node
I20260929 01:07:41.726326 9998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:41.727654 9998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:41.728919 9998 hybrid_clock.cc:648] HybridClock initialized: now 1790644061728883 us; error 44 us; skew 500 ppm
I20260929 01:07:41.729265 9998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:41 dist-test-slave-41cl krb5kdc[9985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644061, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:41.738351 9998 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:07:41.739957 9998 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:41.740079 9998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:41.740377 9998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:41.743292 9998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "d337363ab0a745c09580e7e3f814c59e"
format_stamp: "Formatted at 2026-09-29 01:07:41 on dist-test-slave-41cl"
server_key: "a8743944b43fadcba3ed08b75c8fb544"
server_key_iv: "c1e839f273a0fcc059e8e1834bb22edf"
server_key_version: "encryptionkey@0"
I20260929 01:07:41.744083 9998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "d337363ab0a745c09580e7e3f814c59e"
format_stamp: "Formatted at 2026-09-29 01:07:41 on dist-test-slave-41cl"
server_key: "a8743944b43fadcba3ed08b75c8fb544"
server_key_iv: "c1e839f273a0fcc059e8e1834bb22edf"
server_key_version: "encryptionkey@0"
I20260929 01:07:41.748907 9998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260929 01:07:41.752152 10012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:41.753449 9998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:07:41.753633 9998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "d337363ab0a745c09580e7e3f814c59e"
format_stamp: "Formatted at 2026-09-29 01:07:41 on dist-test-slave-41cl"
server_key: "a8743944b43fadcba3ed08b75c8fb544"
server_key_iv: "c1e839f273a0fcc059e8e1834bb22edf"
server_key_version: "encryptionkey@0"
I20260929 01:07:41.753849 9998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:41.774699 9998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:41.779096 9998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:41.779377 9998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:41.798642 9998 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:33565
I20260929 01:07:41.798616 10063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:33565 every 8 connection(s)
I20260929 01:07:41.799649 9998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:07:41.803285 10064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:41.805749 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 9998
I20260929 01:07:41.805979 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:07:41.806741 5915 external_mini_cluster.cc:1428] Setting key 825e136e9e1587e189c7229d76a59f6e
I20260929 01:07:41.815655 10064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e: Bootstrap starting.
I20260929 01:07:41.819500 10064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:41.820827 10064 log.cc:824] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:07:41 dist-test-slave-41cl krb5kdc[9985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:41.823815 10064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e: No bootstrap required, opened a new log
I20260929 01:07:41.830199 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:41.808923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.5.198.254:33565)
0929 01:07:41.809829 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:41.809844 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:41.809887 (+ 43us) server_negotiation.cc:407] Connection header received
0929 01:07:41.811563 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:41.811584 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:41.811978 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:41.812320 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:41.814171 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:41.815448 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:41.817204 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:41.817893 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:41.818044 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:41.823011 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:41.823019 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:41.823033 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:41.823065 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:41.826134 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:41.826895 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:41.826901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:41.826906 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:41.827064 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:41.827457 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:41.827461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:41.827463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:41.827948 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:41.828154 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:41.828773 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 01:07:41.829198 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":457,"threads_started":1}
I20260929 01:07:41.832749 10064 raft_consensus.cc:348] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d337363ab0a745c09580e7e3f814c59e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33565 } }
I20260929 01:07:41.833129 10064 raft_consensus.cc:374] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:41.833249 10064 raft_consensus.cc:729] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d337363ab0a745c09580e7e3f814c59e, State: Initialized, Role: FOLLOWER
I20260929 01:07:41.833756 10064 consensus_queue.cc:260] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d337363ab0a745c09580e7e3f814c59e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33565 } }
I20260929 01:07:41.833956 10064 raft_consensus.cc:388] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:07:41.834038 10064 raft_consensus.cc:482] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:07:41.834147 10064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:41.836404 10064 raft_consensus.cc:504] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d337363ab0a745c09580e7e3f814c59e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33565 } }
I20260929 01:07:41.836913 10064 leader_election.cc:302] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d337363ab0a745c09580e7e3f814c59e; no voters:
I20260929 01:07:41.837450 10064 leader_election.cc:288] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:07:41.837627 10069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:41.838339 10069 raft_consensus.cc:686] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [term 1 LEADER]: Becoming Leader. State: Replica: d337363ab0a745c09580e7e3f814c59e, State: Running, Role: LEADER
I20260929 01:07:41.838882 10069 consensus_queue.cc:237] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d337363ab0a745c09580e7e3f814c59e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33565 } }
I20260929 01:07:41.839520 10064 sys_catalog.cc:564] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:41.841684 10071 sys_catalog.cc:455] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d337363ab0a745c09580e7e3f814c59e. Latest consensus state: current_term: 1 leader_uuid: "d337363ab0a745c09580e7e3f814c59e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d337363ab0a745c09580e7e3f814c59e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33565 } } }
I20260929 01:07:41.841702 10070 sys_catalog.cc:455] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d337363ab0a745c09580e7e3f814c59e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d337363ab0a745c09580e7e3f814c59e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33565 } } }
I20260929 01:07:41.842108 10070 sys_catalog.cc:458] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:41.842111 10071 sys_catalog.cc:458] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:41.843034 10074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:07:41.848384 10074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:07:41.862591 10074 catalog_manager.cc:1347] Generated new cluster ID: 72d621cf149b40a88a88deb1554eafc3
I20260929 01:07:41.862733 10074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:07:41.902398 10074 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:07:41.903709 10074 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:07:41.915261 10074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e: Generated new TSK 0
I20260929 01:07:41.916247 10074 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:07:41.968186 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33565
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:40661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:42.293092 10090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:42.293339 10090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:42.293432 10090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:42.300688 10090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:42.300781 10090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:42.300894 10090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:07:42.310271 10090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:42.311429 10090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:42.312716 10090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:42.325589 10095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:42.325589 10096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:42.325623 10098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:42.326826 10090 server_base.cc:1034] running on GCE node
I20260929 01:07:42.327440 10090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:42.328359 10090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:42.329582 10090 hybrid_clock.cc:648] HybridClock initialized: now 1790644062329525 us; error 54 us; skew 500 ppm
I20260929 01:07:42.329900 10090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:42 dist-test-slave-41cl krb5kdc[9985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:42.333446 10090 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:07:42.334615 10090 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:42.334750 10090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:42.335065 10090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:42.337952 10090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "5de9db4dfc304c4f9da7f0a37d36e444"
format_stamp: "Formatted at 2026-09-29 01:07:42 on dist-test-slave-41cl"
server_key: "3709e31f2bf5b55b37de312d21b1bb80"
server_key_iv: "5e49a1908a577fd704d13d0c140fee91"
server_key_version: "encryptionkey@0"
I20260929 01:07:42.338779 10090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "5de9db4dfc304c4f9da7f0a37d36e444"
format_stamp: "Formatted at 2026-09-29 01:07:42 on dist-test-slave-41cl"
server_key: "3709e31f2bf5b55b37de312d21b1bb80"
server_key_iv: "5e49a1908a577fd704d13d0c140fee91"
server_key_version: "encryptionkey@0"
I20260929 01:07:42.344026 10090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 01:07:42.347571 10104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:42.348883 10090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:07:42.349021 10090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "5de9db4dfc304c4f9da7f0a37d36e444"
format_stamp: "Formatted at 2026-09-29 01:07:42 on dist-test-slave-41cl"
server_key: "3709e31f2bf5b55b37de312d21b1bb80"
server_key_iv: "5e49a1908a577fd704d13d0c140fee91"
server_key_version: "encryptionkey@0"
I20260929 01:07:42.349192 10090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:42.371261 10090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:42.374920 10090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:42.375144 10090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:42.375985 10090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:42.377348 10090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:42.377471 10090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:42.377594 10090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:42.377682 10090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:42.410274 10090 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:45285
I20260929 01:07:42.410354 10216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:45285 every 8 connection(s)
I20260929 01:07:42.411130 10090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:07:42.417085 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 10090
I20260929 01:07:42.417325 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:07:42.418012 5915 external_mini_cluster.cc:1428] Setting key 1d23c93501df9f711df41b070b9b91aa
Sep 29 01:07:42 dist-test-slave-41cl krb5kdc[9985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:42.434396 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:42.413788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:34847 (local address 127.5.198.254:33565)
0929 01:07:42.414349 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:42.414353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:42.415252 (+ 899us) server_negotiation.cc:407] Connection header received
0929 01:07:42.417150 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:42.417155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:42.417238 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:42.417362 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:42.418969 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:42.420137 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:42.422673 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:42.423119 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:42.423226 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:42.427485 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:42.427494 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:42.427502 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:42.427546 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:42.431015 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:42.431799 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:42.431807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:42.431814 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:42.431962 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:42.432346 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:42.432354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:42.432360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:42.432780 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:42.432931 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:42.433629 (+ 698us) server_negotiation.cc:299] Negotiation successful
0929 01:07:42.433880 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":269,"threads_started":1}
I20260929 01:07:42.435024 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:42.414385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33565 (local address 127.5.198.193:34847)
0929 01:07:42.415110 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:42.415147 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:42.416889 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:42.417621 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:42.417636 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:42.418199 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:42.418784 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:42.418807 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:42.420287 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:42.420295 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:42.422496 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:42.422514 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:42.423344 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:42.423353 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:42.423670 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:42.424405 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:42.424440 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:42.427256 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:42.431154 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:42.431160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:42.431172 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:42.431663 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:42.432067 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:42.432070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:42.432073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:42.432231 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:42.432903 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:42.432909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:42.433251 (+ 342us) client_negotiation.cc:769] Sending connection context
0929 01:07:42.433481 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 01:07:42.433851 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":295,"threads_started":1}
I20260929 01:07:42.436815 10217 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33565
I20260929 01:07:42.437130 10217 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:42.437752 10217 heartbeater.cc:511] Master 127.5.198.254:33565 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
I20260929 01:07:42.439903 10029 ts_manager.cc:194] Registered new tserver with Master: 5de9db4dfc304c4f9da7f0a37d36e444 (127.5.198.193:45285)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
I20260929 01:07:42.441653 10029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:34847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:07:42.464131 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33565
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:40661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:42.783280 10223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:42.783586 10223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:42.783694 10223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:42.791291 10223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:42.791386 10223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:42.791496 10223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:07:42.800890 10223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:42.802138 10223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:42.803388 10223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:42.816010 10228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:42.816041 10229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:42.816731 10231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:42.817070 10223 server_base.cc:1034] running on GCE node
I20260929 01:07:42.817679 10223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:42.818540 10223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:42.819780 10223 hybrid_clock.cc:648] HybridClock initialized: now 1790644062819737 us; error 63 us; skew 500 ppm
I20260929 01:07:42.820113 10223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:42 dist-test-slave-41cl krb5kdc[9985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:42.823076 10223 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:07:42.824155 10223 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:42.824286 10223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:42.824668 10223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:42.827491 10223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "574099d76c804a13826f4236bb1a9c97"
format_stamp: "Formatted at 2026-09-29 01:07:42 on dist-test-slave-41cl"
server_key: "0fbf54d7d58ce4b3ed7b5657630cf4ef"
server_key_iv: "48f919578a284aa681ee24b280f332f8"
server_key_version: "encryptionkey@0"
I20260929 01:07:42.828433 10223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "574099d76c804a13826f4236bb1a9c97"
format_stamp: "Formatted at 2026-09-29 01:07:42 on dist-test-slave-41cl"
server_key: "0fbf54d7d58ce4b3ed7b5657630cf4ef"
server_key_iv: "48f919578a284aa681ee24b280f332f8"
server_key_version: "encryptionkey@0"
I20260929 01:07:42.833488 10223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:07:42.836834 10237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:42.838135 10223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:07:42.838315 10223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "574099d76c804a13826f4236bb1a9c97"
format_stamp: "Formatted at 2026-09-29 01:07:42 on dist-test-slave-41cl"
server_key: "0fbf54d7d58ce4b3ed7b5657630cf4ef"
server_key_iv: "48f919578a284aa681ee24b280f332f8"
server_key_version: "encryptionkey@0"
I20260929 01:07:42.838526 10223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:42.864228 10223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:42.868426 10223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:42.868767 10223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:42.869758 10223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:42.871137 10223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:42.871255 10223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:42.871379 10223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:42.871476 10223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:42.904904 10223 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:45499
I20260929 01:07:42.904991 10349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:45499 every 8 connection(s)
I20260929 01:07:42.905800 10223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:07:42.911841 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 10223
I20260929 01:07:42.912060 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:07:42.912542 5915 external_mini_cluster.cc:1428] Setting key 25957efdffa6ce99c7517c7d4926dec5
Sep 29 01:07:42 dist-test-slave-41cl krb5kdc[9985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:42.925779 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:42.908417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:50327 (local address 127.5.198.254:33565)
0929 01:07:42.908647 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:42.908652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:42.909790 (+ 1138us) server_negotiation.cc:407] Connection header received
0929 01:07:42.911596 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:42.911601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:42.911686 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:42.911796 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:42.913566 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:42.914282 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:42.915808 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:42.916232 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:42.916360 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:42.919686 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:42.919691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:42.919694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:42.919723 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:42.922541 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:42.923478 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:42.923482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:42.923484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:42.923611 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:42.924122 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:42.924125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:42.924127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:42.924381 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:42.924559 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:42.925378 (+ 819us) server_negotiation.cc:299] Negotiation successful
0929 01:07:42.925534 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260929 01:07:42.927314 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:42.908919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33565 (local address 127.5.198.194:50327)
0929 01:07:42.909627 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:42.909672 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:42.911309 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:42.912056 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:42.912065 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:42.912690 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:42.913385 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:42.913405 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:42.914434 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:42.914440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:42.915661 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:42.915673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:42.916505 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:42.916517 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:42.916727 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:42.917434 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:42.917455 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:42.919469 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:42.922692 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:42.922704 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:42.922722 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:42.923340 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:42.923738 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:42.923746 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:42.923753 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:42.923993 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:42.924533 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:42.924544 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:42.924995 (+ 451us) client_negotiation.cc:769] Sending connection context
0929 01:07:42.925276 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 01:07:42.925824 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":276,"threads_started":1}
I20260929 01:07:42.928884 10350 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33565
I20260929 01:07:42.929178 10350 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:42.929798 10350 heartbeater.cc:511] Master 127.5.198.254:33565 requested a full tablet report, sending...
I20260929 01:07:42.931248 10029 ts_manager.cc:194] Registered new tserver with Master: 574099d76c804a13826f4236bb1a9c97 (127.5.198.194:45499)
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
I20260929 01:07:42.932049 10029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:50327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:07:42.956135 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33565
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:40661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:43.272835 10355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:43.273095 10355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:43.273191 10355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:43.280270 10355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:43.280369 10355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:43.280511 10355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:07:43.291324 10355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:43.292603 10355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:43.293816 10355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:43.308962 10361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:43.309128 10363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:43.309470 10360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:43.311645 10355 server_base.cc:1034] running on GCE node
I20260929 01:07:43.312244 10355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:43.313652 10355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:43.314877 10355 hybrid_clock.cc:648] HybridClock initialized: now 1790644063314811 us; error 37 us; skew 500 ppm
I20260929 01:07:43.315197 10355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:43 dist-test-slave-41cl krb5kdc[9985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644063, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:43.319316 10355 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:07:43.320880 10355 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:43.321008 10355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:43.321375 10355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:43.324242 10355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "f0f0035eff334cb896123ed602a62dfb"
format_stamp: "Formatted at 2026-09-29 01:07:43 on dist-test-slave-41cl"
server_key: "9ecb119b210fe2c518589e5129d07551"
server_key_iv: "3b260e02be8cd2412d93b4a23a38ee63"
server_key_version: "encryptionkey@0"
I20260929 01:07:43.325217 10355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "f0f0035eff334cb896123ed602a62dfb"
format_stamp: "Formatted at 2026-09-29 01:07:43 on dist-test-slave-41cl"
server_key: "9ecb119b210fe2c518589e5129d07551"
server_key_iv: "3b260e02be8cd2412d93b4a23a38ee63"
server_key_version: "encryptionkey@0"
I20260929 01:07:43.332199 10355 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 01:07:43.336920 10369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:43.338824 10355 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:07:43.339008 10355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "f0f0035eff334cb896123ed602a62dfb"
format_stamp: "Formatted at 2026-09-29 01:07:43 on dist-test-slave-41cl"
server_key: "9ecb119b210fe2c518589e5129d07551"
server_key_iv: "3b260e02be8cd2412d93b4a23a38ee63"
server_key_version: "encryptionkey@0"
I20260929 01:07:43.339272 10355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:43.356935 10355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:43.361132 10355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:43.361392 10355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:43.362350 10355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:43.364224 10355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:43.364310 10355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:43.364423 10355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:43.364524 10355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 01:07:43.397516 10355 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:36595
I20260929 01:07:43.397599 10481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:36595 every 8 connection(s)
I20260929 01:07:43.398351 10355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:07:43.404569 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 10355
I20260929 01:07:43.404799 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:07:43.405285 5915 external_mini_cluster.cc:1428] Setting key b4e13bb10b25c8ef3272b47b03fa5f7b
Sep 29 01:07:43 dist-test-slave-41cl krb5kdc[9985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644063, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:43.418671 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.400766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:43803 (local address 127.5.198.254:33565)
0929 01:07:43.400971 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:43.400977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:43.402060 (+ 1083us) server_negotiation.cc:407] Connection header received
0929 01:07:43.403926 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:43.403931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:43.404069 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:43.404242 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:43.405885 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.407010 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.408870 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.409318 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.409432 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.413192 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:43.413197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:43.413200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:43.413229 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:43.415788 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:43.416460 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:43.416484 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:43.416502 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:43.416628 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:43.417028 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:43.417033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:43.417037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:43.417293 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:43.417446 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:43.418147 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 01:07:43.418342 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:07:43.419638 10484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.401235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33565 (local address 127.5.198.195:43803)
0929 01:07:43.401906 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:43.401948 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:43.403604 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:43.404387 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:43.404394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:43.405055 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:43.405648 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:43.405670 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.407170 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.407174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:43.408688 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:43.408700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.409521 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.409526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:43.409699 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.410436 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:43.410469 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:43.412972 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:43.415918 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:43.415926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:43.415939 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:43.416334 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:43.416737 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:43.416743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:43.416746 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:43.416909 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:43.417418 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:43.417425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:43.417756 (+ 331us) client_negotiation.cc:769] Sending connection context
0929 01:07:43.418048 (+ 292us) client_negotiation.cc:240] Negotiation successful
0929 01:07:43.418523 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":267,"threads_started":1}
I20260929 01:07:43.421077 10482 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33565
I20260929 01:07:43.421356 10482 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:43.421943 10482 heartbeater.cc:511] Master 127.5.198.254:33565 requested a full tablet report, sending...
I20260929 01:07:43.423373 10029 ts_manager.cc:194] Registered new tserver with Master: f0f0035eff334cb896123ed602a62dfb (127.5.198.195:36595)
I20260929 01:07:43.424331 10029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:43803
I20260929 01:07:43.433290 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:07:43.444653 10217 heartbeater.cc:503] Master 127.5.198.254:33565 was elected leader, sending a full tablet report...
I20260929 01:07:43.450217 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.437730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.5.198.254:33565)
0929 01:07:43.437925 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:43.437929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:43.438400 (+ 471us) server_negotiation.cc:407] Connection header received
0929 01:07:43.438522 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:43.438526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:43.438582 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:43.438661 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:43.439932 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.440671 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.441960 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.442447 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.442542 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.445160 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:43.445168 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:43.445173 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:43.445215 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:43.447946 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:43.448586 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:43.448590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:43.448592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:43.448706 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:43.449038 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:43.449041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:43.449043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:43.449312 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:43.449484 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:43.449736 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 01:07:43.449874 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:07:43.453920 10029 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:07:43.461153 10029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:07:43.484302 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.473828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.5.198.193:45285)
0929 01:07:43.474448 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:43.474459 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:43.477042 (+ 2583us) server_negotiation.cc:407] Connection header received
0929 01:07:43.477233 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:43.477246 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:43.477430 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:43.477628 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:43.478150 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.479570 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.481785 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.483119 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.483201 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.483354 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:43.483822 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 01:07:43.483970 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":300,"threads_started":1}
I20260929 01:07:43.484328 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.473386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:45285 (local address 127.0.0.1:39996)
0929 01:07:43.476930 (+ 3544us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:43.476943 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:43.477068 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:43.477628 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:43.477635 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:43.477669 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:43.477991 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:43.477999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.479692 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.479699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:43.481586 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:43.481602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.483481 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.483485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:43.483612 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.483645 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:07:43.483774 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 01:07:43.483963 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3422,"thread_start_us":3286,"threads_started":1}
I20260929 01:07:43.488896 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.472956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:36595 (local address 127.0.0.1:38398)
0929 01:07:43.475711 (+ 2755us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:43.475725 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:43.475840 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:43.481542 (+ 5702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:43.481547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:43.481573 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:43.482045 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:43.482059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.484239 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.484249 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:43.486142 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:43.486157 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.487945 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.487952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:43.488144 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.488195 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 01:07:43.488299 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 01:07:43.488430 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2640,"mutex_wait_us":1,"thread_start_us":2527,"threads_started":1}
I20260929 01:07:43.489106 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.473635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38398 (local address 127.5.198.195:36595)
0929 01:07:43.480985 (+ 7350us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:43.480994 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:43.481025 (+ 31us) server_negotiation.cc:407] Connection header received
0929 01:07:43.481112 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:43.481119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:43.481339 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:43.481523 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:43.482306 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.484084 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.486299 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.487711 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.487840 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.488052 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:43.488588 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 01:07:43.488734 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7164,"thread_start_us":7028,"threads_started":1}
I20260929 01:07:43.489362 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.472790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:45499 (local address 127.0.0.1:52614)
0929 01:07:43.474966 (+ 2176us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:43.475010 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:43.475200 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:43.481990 (+ 6790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:43.481997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:43.482034 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:43.482508 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:43.482520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.484731 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.484737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:43.486458 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:43.486471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.488242 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.488249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:43.488426 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.488449 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 01:07:43.488588 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 01:07:43.488714 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1883,"thread_start_us":1687,"threads_started":1}
I20260929 01:07:43.489969 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.472768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.5.198.194:45499)
0929 01:07:43.481365 (+ 8597us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:43.481372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:43.481394 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:07:43.481499 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:43.481507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:43.481735 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:43.482729 (+ 994us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:43.482972 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.484586 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.486635 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.488035 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.488174 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.488358 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:43.489302 (+ 944us) server_negotiation.cc:299] Negotiation successful
0929 01:07:43.489657 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8434,"thread_start_us":8334,"threads_started":1}
I20260929 01:07:43.490290 10152 tablet_service.cc:1467] Processing CreateTablet for tablet 904d42ee29994d21bc5235aa5f495157 (DEFAULT_TABLE table=test-table [id=bf0c665fc7f74c36b92eecdfd348dd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:43.492266 10152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904d42ee29994d21bc5235aa5f495157. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:43.493005 10285 tablet_service.cc:1467] Processing CreateTablet for tablet 904d42ee29994d21bc5235aa5f495157 (DEFAULT_TABLE table=test-table [id=bf0c665fc7f74c36b92eecdfd348dd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:43.492901 10417 tablet_service.cc:1467] Processing CreateTablet for tablet 904d42ee29994d21bc5235aa5f495157 (DEFAULT_TABLE table=test-table [id=bf0c665fc7f74c36b92eecdfd348dd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:43.493929 10285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904d42ee29994d21bc5235aa5f495157. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:43.494261 10417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904d42ee29994d21bc5235aa5f495157. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:43.505131 10500 tablet_bootstrap.cc:492] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97: Bootstrap starting.
I20260929 01:07:43.508068 10500 tablet_bootstrap.cc:654] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:43.509135 10501 tablet_bootstrap.cc:492] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb: Bootstrap starting.
I20260929 01:07:43.509339 10500 log.cc:824] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:43.509992 10502 tablet_bootstrap.cc:492] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444: Bootstrap starting.
I20260929 01:07:43.512269 10500 tablet_bootstrap.cc:492] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97: No bootstrap required, opened a new log
I20260929 01:07:43.512615 10500 ts_tablet_manager.cc:1397] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260929 01:07:43.513527 10501 tablet_bootstrap.cc:654] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:43.513579 10502 tablet_bootstrap.cc:654] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:43.514750 10501 log.cc:824] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:43.514750 10502 log.cc:824] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:43.517537 10501 tablet_bootstrap.cc:492] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb: No bootstrap required, opened a new log
I20260929 01:07:43.517784 10501 ts_tablet_manager.cc:1397] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 01:07:43.518036 10502 tablet_bootstrap.cc:492] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444: No bootstrap required, opened a new log
I20260929 01:07:43.518308 10502 ts_tablet_manager.cc:1397] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 01:07:43.524148 10500 raft_consensus.cc:348] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574099d76c804a13826f4236bb1a9c97" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45499 } } peers { permanent_uuid: "f0f0035eff334cb896123ed602a62dfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36595 } } peers { permanent_uuid: "5de9db4dfc304c4f9da7f0a37d36e444" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45285 } }
I20260929 01:07:43.524647 10500 raft_consensus.cc:374] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:43.524756 10500 raft_consensus.cc:729] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 574099d76c804a13826f4236bb1a9c97, State: Initialized, Role: FOLLOWER
I20260929 01:07:43.525341 10501 raft_consensus.cc:348] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574099d76c804a13826f4236bb1a9c97" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45499 } } peers { permanent_uuid: "f0f0035eff334cb896123ed602a62dfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36595 } } peers { permanent_uuid: "5de9db4dfc304c4f9da7f0a37d36e444" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45285 } }
I20260929 01:07:43.525429 10500 consensus_queue.cc:260] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574099d76c804a13826f4236bb1a9c97" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45499 } } peers { permanent_uuid: "f0f0035eff334cb896123ed602a62dfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36595 } } peers { permanent_uuid: "5de9db4dfc304c4f9da7f0a37d36e444" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45285 } }
I20260929 01:07:43.525653 10501 raft_consensus.cc:374] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:43.525745 10501 raft_consensus.cc:729] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f0035eff334cb896123ed602a62dfb, State: Initialized, Role: FOLLOWER
I20260929 01:07:43.526423 10501 consensus_queue.cc:260] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574099d76c804a13826f4236bb1a9c97" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45499 } } peers { permanent_uuid: "f0f0035eff334cb896123ed602a62dfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36595 } } peers { permanent_uuid: "5de9db4dfc304c4f9da7f0a37d36e444" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45285 } }
I20260929 01:07:43.526573 10350 heartbeater.cc:503] Master 127.5.198.254:33565 was elected leader, sending a full tablet report...
I20260929 01:07:43.527439 10482 heartbeater.cc:503] Master 127.5.198.254:33565 was elected leader, sending a full tablet report...
I20260929 01:07:43.528097 10501 ts_tablet_manager.cc:1428] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260929 01:07:43.529156 10500 ts_tablet_manager.cc:1428] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260929 01:07:43.529419 10502 raft_consensus.cc:348] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574099d76c804a13826f4236bb1a9c97" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45499 } } peers { permanent_uuid: "f0f0035eff334cb896123ed602a62dfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36595 } } peers { permanent_uuid: "5de9db4dfc304c4f9da7f0a37d36e444" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45285 } }
I20260929 01:07:43.529863 10502 raft_consensus.cc:374] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:43.529976 10502 raft_consensus.cc:729] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de9db4dfc304c4f9da7f0a37d36e444, State: Initialized, Role: FOLLOWER
I20260929 01:07:43.530582 10502 consensus_queue.cc:260] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574099d76c804a13826f4236bb1a9c97" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45499 } } peers { permanent_uuid: "f0f0035eff334cb896123ed602a62dfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36595 } } peers { permanent_uuid: "5de9db4dfc304c4f9da7f0a37d36e444" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45285 } }
I20260929 01:07:43.532388 10502 ts_tablet_manager.cc:1428] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260929 01:07:43.552985 10506 raft_consensus.cc:482] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:07:43.553246 10506 raft_consensus.cc:504] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574099d76c804a13826f4236bb1a9c97" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45499 } } peers { permanent_uuid: "f0f0035eff334cb896123ed602a62dfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36595 } } peers { permanent_uuid: "5de9db4dfc304c4f9da7f0a37d36e444" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45285 } }
I20260929 01:07:43.554980 10506 leader_election.cc:288] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0f0035eff334cb896123ed602a62dfb (127.5.198.195:36595), 5de9db4dfc304c4f9da7f0a37d36e444 (127.5.198.193:45285)
I20260929 01:07:43.561686 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.555555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:38097 (local address 127.5.198.193:45285)
0929 01:07:43.555738 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:43.555744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:43.556374 (+ 630us) server_negotiation.cc:407] Connection header received
0929 01:07:43.556567 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:43.556572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:43.556644 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:43.556740 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:43.557409 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.558184 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.559995 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.560741 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.560866 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.560915 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:43.561310 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 01:07:43.561435 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 01:07:43.561874 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.555447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:45285 (local address 127.5.198.194:38097)
0929 01:07:43.556219 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:43.556238 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:43.556394 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:43.556785 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:43.556788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:43.556852 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:43.557221 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:43.557234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.558384 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.558390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:43.559835 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:43.559846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.560947 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.560954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:43.561151 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.561186 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 01:07:43.561255 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 01:07:43.561384 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":3,"spinlock_wait_cycles":14720,"thread_start_us":457,"threads_started":1}
I20260929 01:07:43.561890 10509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.555177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:36595 (local address 127.5.198.194:53763)
0929 01:07:43.556060 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:43.556081 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:43.556226 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:43.556586 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:43.556590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:43.556613 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:43.557044 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:43.557053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.558180 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.558185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:43.559837 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:43.559848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.561088 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.561093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:43.561195 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.561216 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:07:43.561328 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 01:07:43.561451 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":73,"thread_start_us":445,"threads_started":1}
I20260929 01:07:43.561715 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.555238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:53763 (local address 127.5.198.195:36595)
0929 01:07:43.555436 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:43.555444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:43.556194 (+ 750us) server_negotiation.cc:407] Connection header received
0929 01:07:43.556370 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:43.556377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:43.556457 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:43.556575 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:43.557190 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.558060 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.560021 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.560877 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.561036 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.561108 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:43.561363 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 01:07:43.561479 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:07:43.562927 10437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904d42ee29994d21bc5235aa5f495157" candidate_uuid: "574099d76c804a13826f4236bb1a9c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f0035eff334cb896123ed602a62dfb" is_pre_election: true
I20260929 01:07:43.562901 10172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904d42ee29994d21bc5235aa5f495157" candidate_uuid: "574099d76c804a13826f4236bb1a9c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de9db4dfc304c4f9da7f0a37d36e444" is_pre_election: true
I20260929 01:07:43.563436 10437 raft_consensus.cc:2454] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 574099d76c804a13826f4236bb1a9c97 in term 0.
I20260929 01:07:43.563436 10172 raft_consensus.cc:2454] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 574099d76c804a13826f4236bb1a9c97 in term 0.
I20260929 01:07:43.564126 10240 leader_election.cc:302] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 574099d76c804a13826f4236bb1a9c97, f0f0035eff334cb896123ed602a62dfb; no voters:
I20260929 01:07:43.564514 10506 raft_consensus.cc:2781] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:07:43.564675 10506 raft_consensus.cc:482] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:07:43.564792 10506 raft_consensus.cc:3037] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:43.566865 10506 raft_consensus.cc:504] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574099d76c804a13826f4236bb1a9c97" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45499 } } peers { permanent_uuid: "f0f0035eff334cb896123ed602a62dfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36595 } } peers { permanent_uuid: "5de9db4dfc304c4f9da7f0a37d36e444" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45285 } }
I20260929 01:07:43.567430 10506 leader_election.cc:288] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [CANDIDATE]: Term 1 election: Requested vote from peers f0f0035eff334cb896123ed602a62dfb (127.5.198.195:36595), 5de9db4dfc304c4f9da7f0a37d36e444 (127.5.198.193:45285)
I20260929 01:07:43.567860 10437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904d42ee29994d21bc5235aa5f495157" candidate_uuid: "574099d76c804a13826f4236bb1a9c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f0035eff334cb896123ed602a62dfb"
I20260929 01:07:43.567876 10172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904d42ee29994d21bc5235aa5f495157" candidate_uuid: "574099d76c804a13826f4236bb1a9c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de9db4dfc304c4f9da7f0a37d36e444"
I20260929 01:07:43.568051 10437 raft_consensus.cc:3037] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:43.568065 10172 raft_consensus.cc:3037] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:43.570164 10437 raft_consensus.cc:2454] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 574099d76c804a13826f4236bb1a9c97 in term 1.
I20260929 01:07:43.570166 10172 raft_consensus.cc:2454] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 574099d76c804a13826f4236bb1a9c97 in term 1.
I20260929 01:07:43.570533 10240 leader_election.cc:302] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 574099d76c804a13826f4236bb1a9c97, f0f0035eff334cb896123ed602a62dfb; no voters:
I20260929 01:07:43.570807 10506 raft_consensus.cc:2781] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:43.571233 10506 raft_consensus.cc:686] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [term 1 LEADER]: Becoming Leader. State: Replica: 574099d76c804a13826f4236bb1a9c97, State: Running, Role: LEADER
I20260929 01:07:43.571676 10506 consensus_queue.cc:237] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574099d76c804a13826f4236bb1a9c97" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45499 } } peers { permanent_uuid: "f0f0035eff334cb896123ed602a62dfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36595 } } peers { permanent_uuid: "5de9db4dfc304c4f9da7f0a37d36e444" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45285 } }
I20260929 01:07:43.576288 10028 catalog_manager.cc:5495] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 574099d76c804a13826f4236bb1a9c97 (127.5.198.194). New cstate: current_term: 1 leader_uuid: "574099d76c804a13826f4236bb1a9c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "574099d76c804a13826f4236bb1a9c97" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0f0035eff334cb896123ed602a62dfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5de9db4dfc304c4f9da7f0a37d36e444" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45285 } health_report { overall_health: UNKNOWN } } }
I20260929 01:07:43.609539 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.604540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.5.198.194:45499)
0929 01:07:43.604730 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:43.604734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:43.604753 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:07:43.604875 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:43.604879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:43.604959 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:43.605095 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:07:43.605400 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.606160 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.607419 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.607884 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.607981 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.608411 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:07:43.609023 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:07:43.609141 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:43.609252 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 01:07:43.609318 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:07:43.617409 10437 raft_consensus.cc:1261] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb [term 1 FOLLOWER]: Refusing update from remote peer 574099d76c804a13826f4236bb1a9c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:43.617489 10172 raft_consensus.cc:1261] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444 [term 1 FOLLOWER]: Refusing update from remote peer 574099d76c804a13826f4236bb1a9c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:43.618325 10511 consensus_queue.cc:1035] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f0035eff334cb896123ed602a62dfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:43.618669 10506 consensus_queue.cc:1035] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de9db4dfc304c4f9da7f0a37d36e444" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:43.629637 10514 mvcc.cc:205] Tried to move back new op lower bound from 7334478084567404544 to 7334478084394094592. Current Snapshot: MvccSnapshot[applied={T|T < 7334478084567404544}]
I20260929 01:07:43.630729 10516 mvcc.cc:205] Tried to move back new op lower bound from 7334478084567404544 to 7334478084394094592. Current Snapshot: MvccSnapshot[applied={T|T < 7334478084567404544}]
I20260929 01:07:43.635015 10515 mvcc.cc:205] Tried to move back new op lower bound from 7334478084567404544 to 7334478084394094592. Current Snapshot: MvccSnapshot[applied={T|T < 7334478084567404544}]
I20260929 01:07:43.644893 10029 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:07:43.645257 10029 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:07:43.648373 10029 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d337363ab0a745c09580e7e3f814c59e: Sending DeleteTablet for 3 replicas of tablet 904d42ee29994d21bc5235aa5f495157
I20260929 01:07:43.649471 10152 tablet_service.cc:1514] Processing DeleteTablet for tablet 904d42ee29994d21bc5235aa5f495157 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:43 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:39996
I20260929 01:07:43.649485 10417 tablet_service.cc:1514] Processing DeleteTablet for tablet 904d42ee29994d21bc5235aa5f495157 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:43 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:38398
I20260929 01:07:43.649524 10285 tablet_service.cc:1514] Processing DeleteTablet for tablet 904d42ee29994d21bc5235aa5f495157 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:43 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:52614
I20260929 01:07:43.652339 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 10090
I20260929 01:07:43.664826 10525 tablet_replica.cc:330] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb: stopping tablet replica
I20260929 01:07:43.665313 10525 raft_consensus.cc:2229] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:07:43.665828 10525 raft_consensus.cc:2258] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:43.669615 10525 ts_tablet_manager.cc:1905] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:43.678251 10525 ts_tablet_manager.cc:1918] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:07:43.678445 10525 log.cc:1196] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/904d42ee29994d21bc5235aa5f495157
I20260929 01:07:43.679015 10525 ts_tablet_manager.cc:1939] T 904d42ee29994d21bc5235aa5f495157 P f0f0035eff334cb896123ed602a62dfb: Deleting consensus metadata
I20260929 01:07:43.680881 10015 catalog_manager.cc:4841] TS f0f0035eff334cb896123ed602a62dfb (127.5.198.195:36595): tablet 904d42ee29994d21bc5235aa5f495157 (table test-table [id=bf0c665fc7f74c36b92eecdfd348dd39]) successfully deleted
I20260929 01:07:43.682988 10524 tablet_replica.cc:330] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97: stopping tablet replica
I20260929 01:07:43.683418 10524 raft_consensus.cc:2229] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:07:43.684029 10524 raft_consensus.cc:2258] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:43.685951 10523 tablet_replica.cc:330] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444: stopping tablet replica
I20260929 01:07:43.686374 10523 raft_consensus.cc:2229] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:07:43.686820 10523 raft_consensus.cc:2258] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:43.687301 10524 ts_tablet_manager.cc:1905] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:43.688711 10523 ts_tablet_manager.cc:1905] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 01:07:43 dist-test-slave-41cl krb5kdc[9985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:07:43.694722 10524 ts_tablet_manager.cc:1918] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:07:43.694897 10524 log.cc:1196] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/904d42ee29994d21bc5235aa5f495157
I20260929 01:07:43.695426 10524 ts_tablet_manager.cc:1939] T 904d42ee29994d21bc5235aa5f495157 P 574099d76c804a13826f4236bb1a9c97: Deleting consensus metadata
I20260929 01:07:43.695919 10523 ts_tablet_manager.cc:1918] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:07:43.696081 10523 log.cc:1196] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/904d42ee29994d21bc5235aa5f495157
I20260929 01:07:43.696621 10523 ts_tablet_manager.cc:1939] T 904d42ee29994d21bc5235aa5f495157 P 5de9db4dfc304c4f9da7f0a37d36e444: Deleting consensus metadata
I20260929 01:07:43.697261 10016 catalog_manager.cc:4841] TS 574099d76c804a13826f4236bb1a9c97 (127.5.198.194:45499): tablet 904d42ee29994d21bc5235aa5f495157 (table test-table [id=bf0c665fc7f74c36b92eecdfd348dd39]) successfully deleted
I20260929 01:07:43.698282 10014 catalog_manager.cc:4841] TS 5de9db4dfc304c4f9da7f0a37d36e444 (127.5.198.193:45285): tablet 904d42ee29994d21bc5235aa5f495157 (table test-table [id=bf0c665fc7f74c36b92eecdfd348dd39]) successfully deleted
I20260929 01:07:43.699682 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.655869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39998 (local address 127.5.198.193:45285)
0929 01:07:43.656082 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:43.656091 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:43.679467 (+ 23376us) server_negotiation.cc:407] Connection header received
0929 01:07:43.679940 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:43.679947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:43.680035 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:43.680111 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:43.682098 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.683377 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.685520 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.686212 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.686335 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.692009 (+ 5674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:43.692025 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:43.692042 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:43.692093 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:43.695964 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:43.697139 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:43.697148 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:43.697151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:43.697308 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:43.697831 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:43.697836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:43.697839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:43.698296 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:43.698640 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:43.699121 (+ 481us) server_negotiation.cc:299] Negotiation successful
0929 01:07:43.699337 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:07:43.700336 10212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:43.789709 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 10090
I20260929 01:07:43.810793 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 10223
Sep 29 01:07:43 dist-test-slave-41cl krb5kdc[9985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:07:43.829576 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.811981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.5.198.194:45499)
0929 01:07:43.812144 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:43.812150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:43.812170 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:07:43.812350 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:43.812355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:43.812445 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:43.812593 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:43.814083 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.815285 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.817119 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.817821 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.817942 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.822972 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:43.822986 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:43.823005 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:43.823056 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:43.826878 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:43.827486 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:43.827495 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:43.827499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:43.827656 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:43.828001 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:43.828006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:43.828009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:43.828441 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:43.828797 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:43.828979 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 01:07:43.829179 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 01:07:43.830374 10345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:43.945744 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 10223
I20260929 01:07:43.964382 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 10355
Sep 29 01:07:43 dist-test-slave-41cl krb5kdc[9985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:07:43.979272 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:43.965268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38406 (local address 127.5.198.195:36595)
0929 01:07:43.965472 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:43.965481 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:43.965502 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:07:43.965709 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:43.965719 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:43.965788 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:43.965928 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:43.967145 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.967882 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.969129 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:43.969527 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:43.969655 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:43.974070 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:43.974083 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:43.974098 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:43.974130 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:43.976691 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:43.977295 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:43.977310 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:43.977313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:43.977438 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:43.977807 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:43.977812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:43.977815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:43.978150 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:43.978435 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:43.978744 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 01:07:43.978988 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:07:43.979893 10477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:44.066937 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 10355
I20260929 01:07:44.084928 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 9998
I20260929 01:07:44.086055 10059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:44.191799 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 9998
2026-09-29T01:07:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4098 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T01:07:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:07:44Z Disabled control of system clock
I20260929 01:07:44.256062 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:44297
--webserver_interface=127.5.198.254
--webserver_port=0
--builtin_ntp_servers=127.5.198.212:35611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:44297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 01:07:44.565701 10534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:44.565981 10534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:44.566066 10534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:44.573055 10534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:44.573154 10534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:44.573190 10534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:44.573236 10534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:44.573278 10534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:07:44.582386 10534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:35611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:44297
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:44297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:44.583658 10534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:44.585423 10534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:44.599399 10540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:44.599512 10539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:44.599488 10542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:44.600593 10534 server_base.cc:1034] running on GCE node
I20260929 01:07:44.601365 10534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:44.602691 10534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:44.603888 10534 hybrid_clock.cc:648] HybridClock initialized: now 1790644064603837 us; error 53 us; skew 500 ppm
I20260929 01:07:44.604260 10534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:07:44.606688 10534 webserver.cc:460] Webserver started at http://127.5.198.254:33755/ using document root <none> and password file <none>
I20260929 01:07:44.607370 10534 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:44.607456 10534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:44.607806 10534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:44.610677 10534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "f4cdeb31c7e64deb979c721427708e58"
format_stamp: "Formatted at 2026-09-29 01:07:44 on dist-test-slave-41cl"
server_key: "2bef4cbc0ca7d086762288984a8aa3a8"
server_key_iv: "781786078763d7f499c756b55e37139f"
server_key_version: "encryptionkey@0"
I20260929 01:07:44.611478 10534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "f4cdeb31c7e64deb979c721427708e58"
format_stamp: "Formatted at 2026-09-29 01:07:44 on dist-test-slave-41cl"
server_key: "2bef4cbc0ca7d086762288984a8aa3a8"
server_key_iv: "781786078763d7f499c756b55e37139f"
server_key_version: "encryptionkey@0"
I20260929 01:07:44.616269 10534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260929 01:07:44.619459 10548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:44.620792 10534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:07:44.620994 10534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "f4cdeb31c7e64deb979c721427708e58"
format_stamp: "Formatted at 2026-09-29 01:07:44 on dist-test-slave-41cl"
server_key: "2bef4cbc0ca7d086762288984a8aa3a8"
server_key_iv: "781786078763d7f499c756b55e37139f"
server_key_version: "encryptionkey@0"
I20260929 01:07:44.621161 10534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:44.626341 10534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:44.626982 10534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:44.627220 10534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:44.644270 10534 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:44297
I20260929 01:07:44.644306 10599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:44297 every 8 connection(s)
I20260929 01:07:44.645514 10534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:07:44.649020 10600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:44.651765 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 10534
I20260929 01:07:44.651963 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:07:44.652784 5915 external_mini_cluster.cc:1428] Setting key 01c56696268dfaac5c08a2b260a08982
I20260929 01:07:44.660833 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:44.654817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60664 (local address 127.5.198.254:44297)
0929 01:07:44.655751 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:44.655761 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:44.655798 (+ 37us) server_negotiation.cc:407] Connection header received
0929 01:07:44.657398 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:44.657419 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:44.657817 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:44.658206 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:44.659282 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:44.659287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:44.659299 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:44.659311 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:44.659365 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:44.659379 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:44.659605 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:44.659733 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 01:07:44.659958 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":336,"threads_started":1}
I20260929 01:07:44.660953 10600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58: Bootstrap starting.
I20260929 01:07:44.664142 10600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:44.665153 10600 log.cc:824] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:44.667532 10600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58: No bootstrap required, opened a new log
I20260929 01:07:44.674616 10600 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cdeb31c7e64deb979c721427708e58" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44297 } }
I20260929 01:07:44.674968 10600 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:44.675073 10600 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4cdeb31c7e64deb979c721427708e58, State: Initialized, Role: FOLLOWER
I20260929 01:07:44.675662 10600 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cdeb31c7e64deb979c721427708e58" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44297 } }
I20260929 01:07:44.675901 10600 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:07:44.676047 10600 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:07:44.676213 10600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:44.678301 10600 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cdeb31c7e64deb979c721427708e58" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44297 } }
I20260929 01:07:44.678764 10600 leader_election.cc:302] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4cdeb31c7e64deb979c721427708e58; no voters:
I20260929 01:07:44.679391 10600 leader_election.cc:288] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:07:44.679529 10605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:44.680313 10605 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [term 1 LEADER]: Becoming Leader. State: Replica: f4cdeb31c7e64deb979c721427708e58, State: Running, Role: LEADER
I20260929 01:07:44.680860 10605 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cdeb31c7e64deb979c721427708e58" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44297 } }
I20260929 01:07:44.681523 10600 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:44.683491 10606 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4cdeb31c7e64deb979c721427708e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cdeb31c7e64deb979c721427708e58" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44297 } } }
I20260929 01:07:44.684007 10606 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:44.683470 10607 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4cdeb31c7e64deb979c721427708e58. Latest consensus state: current_term: 1 leader_uuid: "f4cdeb31c7e64deb979c721427708e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cdeb31c7e64deb979c721427708e58" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44297 } } }
I20260929 01:07:44.684661 10607 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:44.686174 10614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:07:44.691103 10614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:07:44.698843 10614 catalog_manager.cc:1347] Generated new cluster ID: b3b4a589db454da89139f5276d66031e
I20260929 01:07:44.698959 10614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:07:44.723220 10614 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:07:44.723626 10614 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:07:44.732353 10614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58: Generated new TSK 0
I20260929 01:07:44.733107 10614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:07:44.748734 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44297
--builtin_ntp_servers=127.5.198.212:35611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 01:07:45.071975 10624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:45.072235 10624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:45.072382 10624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:45.079538 10624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:45.079633 10624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:45.079803 10624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:07:45.089217 10624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:35611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:45.090480 10624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:45.092232 10624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:45.105145 10630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:45.105171 10632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:45.105796 10629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:45.106117 10624 server_base.cc:1034] running on GCE node
I20260929 01:07:45.106726 10624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:45.107707 10624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:45.108973 10624 hybrid_clock.cc:648] HybridClock initialized: now 1790644065108888 us; error 48 us; skew 500 ppm
I20260929 01:07:45.109305 10624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:07:45.111575 10624 webserver.cc:460] Webserver started at http://127.5.198.193:40389/ using document root <none> and password file <none>
I20260929 01:07:45.112317 10624 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:45.112445 10624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:45.112815 10624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:45.115677 10624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "6397ef0d7bfb492d93357d736f3a72dd"
format_stamp: "Formatted at 2026-09-29 01:07:45 on dist-test-slave-41cl"
server_key: "d235da6d18e1c63de1c8d384d5d9d9ad"
server_key_iv: "3ffc760a051fafa576dd51f154fd1dc9"
server_key_version: "encryptionkey@0"
I20260929 01:07:45.116551 10624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "6397ef0d7bfb492d93357d736f3a72dd"
format_stamp: "Formatted at 2026-09-29 01:07:45 on dist-test-slave-41cl"
server_key: "d235da6d18e1c63de1c8d384d5d9d9ad"
server_key_iv: "3ffc760a051fafa576dd51f154fd1dc9"
server_key_version: "encryptionkey@0"
I20260929 01:07:45.121732 10624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 01:07:45.125294 10638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:45.126639 10624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 01:07:45.126806 10624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "6397ef0d7bfb492d93357d736f3a72dd"
format_stamp: "Formatted at 2026-09-29 01:07:45 on dist-test-slave-41cl"
server_key: "d235da6d18e1c63de1c8d384d5d9d9ad"
server_key_iv: "3ffc760a051fafa576dd51f154fd1dc9"
server_key_version: "encryptionkey@0"
I20260929 01:07:45.127043 10624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:45.132575 10624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:45.133375 10624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:45.133644 10624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:45.134553 10624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:45.135874 10624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:45.136001 10624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:45.136138 10624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:45.136226 10624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:45.169945 10624 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:33979
I20260929 01:07:45.170032 10750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:33979 every 8 connection(s)
I20260929 01:07:45.171234 10624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:07:45.178941 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 10624
I20260929 01:07:45.179147 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:07:45.179349 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:45.173733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:43165 (local address 127.5.198.254:44297)
0929 01:07:45.174157 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:45.174161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:45.175286 (+ 1125us) server_negotiation.cc:407] Connection header received
0929 01:07:45.177043 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:45.177048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:45.177116 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:45.177230 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:45.178576 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:45.178581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:45.178584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:45.178600 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:45.178643 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:45.178655 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:45.178759 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:45.178992 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 01:07:45.179071 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":234,"threads_started":1}
I20260929 01:07:45.179913 5915 external_mini_cluster.cc:1428] Setting key f81ff04732cbec17cbe2f9aefff3f387
I20260929 01:07:45.180528 10754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:45.174167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44297 (local address 127.5.198.193:43165)
0929 01:07:45.175124 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:45.175166 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:45.176784 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:45.177480 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:45.177488 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:45.177574 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:45.178257 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:07:45.178259 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:45.178301 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:07:45.178379 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:45.178765 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:45.178776 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:45.178783 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 01:07:45.179066 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 01:07:45.179334 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":261,"threads_started":1}
I20260929 01:07:45.182147 10751 heartbeater.cc:347] Connected to a master server at 127.5.198.254:44297
I20260929 01:07:45.182448 10751 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:45.183040 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44297
--builtin_ntp_servers=127.5.198.212:35611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 01:07:45.183141 10751 heartbeater.cc:511] Master 127.5.198.254:44297 requested a full tablet report, sending...
I20260929 01:07:45.185387 10565 ts_manager.cc:194] Registered new tserver with Master: 6397ef0d7bfb492d93357d736f3a72dd (127.5.198.193:33979)
W20260929 01:07:45.497182 10755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:45.497478 10755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:45.497650 10755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:45.504645 10755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:45.504770 10755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:45.504945 10755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:07:45.513789 10755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:35611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:45.514952 10755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:45.516641 10755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:45.529567 10760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:45.529603 10761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:45.530030 10763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:45.530797 10755 server_base.cc:1034] running on GCE node
I20260929 01:07:45.531461 10755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:45.532557 10755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:45.533779 10755 hybrid_clock.cc:648] HybridClock initialized: now 1790644065533716 us; error 53 us; skew 500 ppm
I20260929 01:07:45.534173 10755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:07:45.536767 10755 webserver.cc:460] Webserver started at http://127.5.198.194:39863/ using document root <none> and password file <none>
I20260929 01:07:45.537478 10755 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:45.537568 10755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:45.537940 10755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:45.540971 10755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "9b8db306577b4fee8e86297833c402ee"
format_stamp: "Formatted at 2026-09-29 01:07:45 on dist-test-slave-41cl"
server_key: "5af8b03ff85a4aeb705758a85d63e211"
server_key_iv: "35e130c364f8b234f88cdd8e7597f1b9"
server_key_version: "encryptionkey@0"
I20260929 01:07:45.541888 10755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "9b8db306577b4fee8e86297833c402ee"
format_stamp: "Formatted at 2026-09-29 01:07:45 on dist-test-slave-41cl"
server_key: "5af8b03ff85a4aeb705758a85d63e211"
server_key_iv: "35e130c364f8b234f88cdd8e7597f1b9"
server_key_version: "encryptionkey@0"
I20260929 01:07:45.547224 10755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:07:45.550576 10769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:45.551968 10755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:07:45.552165 10755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "9b8db306577b4fee8e86297833c402ee"
format_stamp: "Formatted at 2026-09-29 01:07:45 on dist-test-slave-41cl"
server_key: "5af8b03ff85a4aeb705758a85d63e211"
server_key_iv: "35e130c364f8b234f88cdd8e7597f1b9"
server_key_version: "encryptionkey@0"
I20260929 01:07:45.552434 10755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:45.558202 10755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:45.558998 10755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:45.559294 10755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:45.560261 10755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:45.561703 10755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:45.561833 10755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:45.561983 10755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:45.562080 10755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:45.598070 10755 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:38639
I20260929 01:07:45.598161 10881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:38639 every 8 connection(s)
I20260929 01:07:45.599366 10755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:07:45.602749 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 10755
I20260929 01:07:45.602927 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:07:45.603351 5915 external_mini_cluster.cc:1428] Setting key 70d29a15d27060c15a7d72827749c83b
I20260929 01:07:45.606118 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44297
--builtin_ntp_servers=127.5.198.212:35611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 01:07:45.607837 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:45.602048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:58531 (local address 127.5.198.254:44297)
0929 01:07:45.602255 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:45.602261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:45.603627 (+ 1366us) server_negotiation.cc:407] Connection header received
0929 01:07:45.605586 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:45.605591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:45.605672 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:45.605781 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:45.607129 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:45.607136 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:45.607145 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:45.607160 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:45.607189 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:45.607199 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:45.607314 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:45.607497 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 01:07:45.607560 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:07:45.608904 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:45.602380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44297 (local address 127.5.198.194:58531)
0929 01:07:45.603437 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:45.603496 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:45.605303 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:45.606008 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:45.606016 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:45.606104 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:45.606852 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:07:45.606855 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:45.606889 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:07:45.606971 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:45.607289 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:45.607299 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:45.607307 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 01:07:45.607577 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 01:07:45.607835 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":329,"threads_started":1}
I20260929 01:07:45.610301 10882 heartbeater.cc:347] Connected to a master server at 127.5.198.254:44297
I20260929 01:07:45.610584 10882 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:45.611128 10882 heartbeater.cc:511] Master 127.5.198.254:44297 requested a full tablet report, sending...
I20260929 01:07:45.612659 10565 ts_manager.cc:194] Registered new tserver with Master: 9b8db306577b4fee8e86297833c402ee (127.5.198.194:38639)
W20260929 01:07:45.922724 10885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:45.923045 10885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:45.923184 10885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:45.930570 10885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:45.930661 10885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:45.930765 10885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:07:45.939692 10885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:35611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:45.940953 10885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:45.942605 10885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:45.955642 10893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:45.955698 10890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:45.955652 10891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:45.957909 10885 server_base.cc:1034] running on GCE node
I20260929 01:07:45.958480 10885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:45.959370 10885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:45.960601 10885 hybrid_clock.cc:648] HybridClock initialized: now 1790644065960605 us; error 102 us; skew 500 ppm
I20260929 01:07:45.960968 10885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:07:45.963479 10885 webserver.cc:460] Webserver started at http://127.5.198.195:36673/ using document root <none> and password file <none>
I20260929 01:07:45.964226 10885 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:45.964346 10885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:45.964699 10885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:45.967447 10885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "cb6400349f4b436e8918ce7a3fe67507"
format_stamp: "Formatted at 2026-09-29 01:07:45 on dist-test-slave-41cl"
server_key: "bd90127d923f30e5d917e87a8f161e1d"
server_key_iv: "ef70c8ef006a423e8eff7d7b478da0d8"
server_key_version: "encryptionkey@0"
I20260929 01:07:45.968214 10885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "cb6400349f4b436e8918ce7a3fe67507"
format_stamp: "Formatted at 2026-09-29 01:07:45 on dist-test-slave-41cl"
server_key: "bd90127d923f30e5d917e87a8f161e1d"
server_key_iv: "ef70c8ef006a423e8eff7d7b478da0d8"
server_key_version: "encryptionkey@0"
I20260929 01:07:45.973172 10885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 01:07:45.976589 10899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:45.977867 10885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 01:07:45.978039 10885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "cb6400349f4b436e8918ce7a3fe67507"
format_stamp: "Formatted at 2026-09-29 01:07:45 on dist-test-slave-41cl"
server_key: "bd90127d923f30e5d917e87a8f161e1d"
server_key_iv: "ef70c8ef006a423e8eff7d7b478da0d8"
server_key_version: "encryptionkey@0"
I20260929 01:07:45.978250 10885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:45.983807 10885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:45.984668 10885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:45.984942 10885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:45.985914 10885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:45.987228 10885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:45.987357 10885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:45.987496 10885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:45.987593 10885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:46.019680 10885 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:41219
I20260929 01:07:46.019766 11011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:41219 every 8 connection(s)
I20260929 01:07:46.021044 10885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:07:46.025522 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 10885
I20260929 01:07:46.025683 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:07:46.026213 5915 external_mini_cluster.cc:1428] Setting key 97ba3857b8151acff33dc250a53c3437
I20260929 01:07:46.029592 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.023495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:55641 (local address 127.5.198.254:44297)
0929 01:07:46.023666 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:46.023670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:46.025232 (+ 1562us) server_negotiation.cc:407] Connection header received
0929 01:07:46.027311 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:46.027317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:46.027384 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:46.027499 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:46.028866 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:46.028871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:46.028875 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:46.028896 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:46.028937 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:46.028951 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:46.029107 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:46.029335 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 01:07:46.029381 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:07:46.030591 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.023995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44297 (local address 127.5.198.195:55641)
0929 01:07:46.025045 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:46.025105 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:46.027079 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:46.027714 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:46.027721 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:46.027808 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:46.028594 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:07:46.028597 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:46.028631 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:07:46.028705 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:46.029090 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:46.029100 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:46.029108 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 01:07:46.029429 (+ 321us) client_negotiation.cc:240] Negotiation successful
0929 01:07:46.029612 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":338,"threads_started":1}
I20260929 01:07:46.031842 11012 heartbeater.cc:347] Connected to a master server at 127.5.198.254:44297
I20260929 01:07:46.032120 11012 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:46.032737 11012 heartbeater.cc:511] Master 127.5.198.254:44297 requested a full tablet report, sending...
I20260929 01:07:46.033917 10565 ts_manager.cc:194] Registered new tserver with Master: cb6400349f4b436e8918ce7a3fe67507 (127.5.198.195:41219)
I20260929 01:07:46.041303 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:07:46.047855 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.045539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.5.198.254:44297)
0929 01:07:46.045736 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:46.045743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:46.046196 (+ 453us) server_negotiation.cc:407] Connection header received
0929 01:07:46.046315 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:46.046318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:46.046361 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:46.046473 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:46.047292 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:46.047295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:46.047298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:46.047312 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:46.047341 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:46.047351 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:46.047441 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:46.047524 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 01:07:46.047562 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:07:46.051483 10565 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:07:46.058199 10565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:07:46.088145 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.069465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:38639 (local address 127.0.0.1:53428)
0929 01:07:46.079795 (+ 10330us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:46.079849 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:46.079990 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:46.085092 (+ 5102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:46.085103 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:46.085227 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:46.086601 (+ 1374us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:07:46.086603 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:46.086626 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:07:46.086698 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:46.087360 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:46.087370 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:46.087379 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 01:07:46.087558 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 01:07:46.087644 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10075,"thread_start_us":11751,"threads_started":1}
I20260929 01:07:46.089206 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.070354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53428 (local address 127.5.198.194:38639)
0929 01:07:46.083051 (+ 12697us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:46.083064 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:46.083105 (+ 41us) server_negotiation.cc:407] Connection header received
0929 01:07:46.083192 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:46.083204 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:46.083524 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:46.083706 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:46.086911 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:46.086921 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:46.086935 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:46.086954 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:46.087016 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:46.087043 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:46.087234 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:46.088755 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0929 01:07:46.088860 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12442,"thread_start_us":12265,"threads_started":1}
I20260929 01:07:46.089787 11024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.069877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:41219 (local address 127.0.0.1:48206)
0929 01:07:46.080684 (+ 10807us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:46.080705 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:46.080815 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:46.086809 (+ 5994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:46.086813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:46.086894 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:46.088775 (+ 1881us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:07:46.088778 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:46.088794 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:07:46.088853 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:46.089240 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:46.089244 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:46.089249 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 01:07:46.089369 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 01:07:46.089417 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10613,"thread_start_us":18286,"threads_started":1}
I20260929 01:07:46.089846 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.070729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48206 (local address 127.5.198.195:41219)
0929 01:07:46.086231 (+ 15502us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:46.086239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:46.086276 (+ 37us) server_negotiation.cc:407] Connection header received
0929 01:07:46.086355 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:46.086362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:46.086625 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:46.086804 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:46.089042 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:46.089050 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:46.089067 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:46.089082 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:46.089125 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:46.089143 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:46.089344 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:46.089405 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 01:07:46.089496 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15328,"thread_start_us":15146,"threads_started":1}
I20260929 01:07:46.091720 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.075712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:33979 (local address 127.0.0.1:41100)
0929 01:07:46.088428 (+ 12716us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:46.088447 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:46.088567 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:46.089927 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:46.089934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:46.089983 (+ 49us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:46.090546 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:07:46.090549 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:46.090563 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:07:46.090606 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:46.091305 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:46.091310 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:46.091315 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 01:07:46.091413 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 01:07:46.091461 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12587,"thread_start_us":3659,"threads_started":1}
I20260929 01:07:46.092322 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.069674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.5.198.193:33979)
0929 01:07:46.086291 (+ 16617us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:46.086300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:46.088620 (+ 2320us) server_negotiation.cc:407] Connection header received
0929 01:07:46.088704 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:46.088712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:46.088895 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:46.089031 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:46.090770 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:46.090780 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:46.090791 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:46.090820 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:46.090894 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:46.090917 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:46.091260 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:46.091947 (+ 687us) server_negotiation.cc:299] Negotiation successful
0929 01:07:46.092036 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16457,"thread_start_us":16619,"threads_started":1}
I20260929 01:07:46.093624 10817 tablet_service.cc:1467] Processing CreateTablet for tablet 6504853e0b5f4e5784d91b033e98044e (DEFAULT_TABLE table=test-table [id=acee23f8c6864777abcce759e24668a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:46.093628 10947 tablet_service.cc:1467] Processing CreateTablet for tablet 6504853e0b5f4e5784d91b033e98044e (DEFAULT_TABLE table=test-table [id=acee23f8c6864777abcce759e24668a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:46.094841 10817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6504853e0b5f4e5784d91b033e98044e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:46.094928 10947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6504853e0b5f4e5784d91b033e98044e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:46.095595 10686 tablet_service.cc:1467] Processing CreateTablet for tablet 6504853e0b5f4e5784d91b033e98044e (DEFAULT_TABLE table=test-table [id=acee23f8c6864777abcce759e24668a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:07:46.097012 10686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6504853e0b5f4e5784d91b033e98044e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:46.106173 11030 tablet_bootstrap.cc:492] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee: Bootstrap starting.
I20260929 01:07:46.108966 11031 tablet_bootstrap.cc:492] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507: Bootstrap starting.
I20260929 01:07:46.110505 11032 tablet_bootstrap.cc:492] T 6504853e0b5f4e5784d91b033e98044e P 6397ef0d7bfb492d93357d736f3a72dd: Bootstrap starting.
I20260929 01:07:46.111406 11030 tablet_bootstrap.cc:654] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:46.112411 11031 tablet_bootstrap.cc:654] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:46.112630 11030 log.cc:824] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:46.113351 11032 tablet_bootstrap.cc:654] T 6504853e0b5f4e5784d91b033e98044e P 6397ef0d7bfb492d93357d736f3a72dd: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:46.113646 11031 log.cc:824] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:46.114533 11032 log.cc:824] T 6504853e0b5f4e5784d91b033e98044e P 6397ef0d7bfb492d93357d736f3a72dd: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:46.115450 11030 tablet_bootstrap.cc:492] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee: No bootstrap required, opened a new log
I20260929 01:07:46.115710 11030 ts_tablet_manager.cc:1397] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 01:07:46.116573 11031 tablet_bootstrap.cc:492] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507: No bootstrap required, opened a new log
I20260929 01:07:46.116811 11031 ts_tablet_manager.cc:1397] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260929 01:07:46.117029 11032 tablet_bootstrap.cc:492] T 6504853e0b5f4e5784d91b033e98044e P 6397ef0d7bfb492d93357d736f3a72dd: No bootstrap required, opened a new log
I20260929 01:07:46.117252 11032 ts_tablet_manager.cc:1397] T 6504853e0b5f4e5784d91b033e98044e P 6397ef0d7bfb492d93357d736f3a72dd: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 01:07:46.124154 11032 raft_consensus.cc:348] T 6504853e0b5f4e5784d91b033e98044e P 6397ef0d7bfb492d93357d736f3a72dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8db306577b4fee8e86297833c402ee" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38639 } } peers { permanent_uuid: "6397ef0d7bfb492d93357d736f3a72dd" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 33979 } } peers { permanent_uuid: "cb6400349f4b436e8918ce7a3fe67507" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 41219 } }
I20260929 01:07:46.124593 11032 raft_consensus.cc:374] T 6504853e0b5f4e5784d91b033e98044e P 6397ef0d7bfb492d93357d736f3a72dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:46.124684 11032 raft_consensus.cc:729] T 6504853e0b5f4e5784d91b033e98044e P 6397ef0d7bfb492d93357d736f3a72dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6397ef0d7bfb492d93357d736f3a72dd, State: Initialized, Role: FOLLOWER
I20260929 01:07:46.125270 11032 consensus_queue.cc:260] T 6504853e0b5f4e5784d91b033e98044e P 6397ef0d7bfb492d93357d736f3a72dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8db306577b4fee8e86297833c402ee" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38639 } } peers { permanent_uuid: "6397ef0d7bfb492d93357d736f3a72dd" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 33979 } } peers { permanent_uuid: "cb6400349f4b436e8918ce7a3fe67507" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 41219 } }
I20260929 01:07:46.126276 10751 heartbeater.cc:503] Master 127.5.198.254:44297 was elected leader, sending a full tablet report...
I20260929 01:07:46.126763 11032 ts_tablet_manager.cc:1428] T 6504853e0b5f4e5784d91b033e98044e P 6397ef0d7bfb492d93357d736f3a72dd: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260929 01:07:46.126842 11031 raft_consensus.cc:348] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8db306577b4fee8e86297833c402ee" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38639 } } peers { permanent_uuid: "6397ef0d7bfb492d93357d736f3a72dd" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 33979 } } peers { permanent_uuid: "cb6400349f4b436e8918ce7a3fe67507" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 41219 } }
I20260929 01:07:46.127233 11031 raft_consensus.cc:374] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:46.127345 11031 raft_consensus.cc:729] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6400349f4b436e8918ce7a3fe67507, State: Initialized, Role: FOLLOWER
I20260929 01:07:46.127315 11030 raft_consensus.cc:348] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8db306577b4fee8e86297833c402ee" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38639 } } peers { permanent_uuid: "6397ef0d7bfb492d93357d736f3a72dd" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 33979 } } peers { permanent_uuid: "cb6400349f4b436e8918ce7a3fe67507" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 41219 } }
I20260929 01:07:46.127763 11030 raft_consensus.cc:374] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:46.127856 11030 raft_consensus.cc:729] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8db306577b4fee8e86297833c402ee, State: Initialized, Role: FOLLOWER
I20260929 01:07:46.127899 11031 consensus_queue.cc:260] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8db306577b4fee8e86297833c402ee" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38639 } } peers { permanent_uuid: "6397ef0d7bfb492d93357d736f3a72dd" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 33979 } } peers { permanent_uuid: "cb6400349f4b436e8918ce7a3fe67507" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 41219 } }
I20260929 01:07:46.128455 11030 consensus_queue.cc:260] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8db306577b4fee8e86297833c402ee" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38639 } } peers { permanent_uuid: "6397ef0d7bfb492d93357d736f3a72dd" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 33979 } } peers { permanent_uuid: "cb6400349f4b436e8918ce7a3fe67507" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 41219 } }
I20260929 01:07:46.131665 11031 ts_tablet_manager.cc:1428] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260929 01:07:46.131985 11012 heartbeater.cc:503] Master 127.5.198.254:44297 was elected leader, sending a full tablet report...
I20260929 01:07:46.132565 10882 heartbeater.cc:503] Master 127.5.198.254:44297 was elected leader, sending a full tablet report...
I20260929 01:07:46.133733 11030 ts_tablet_manager.cc:1428] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
W20260929 01:07:46.173401 10752 tablet.cc:2369] T 6504853e0b5f4e5784d91b033e98044e P 6397ef0d7bfb492d93357d736f3a72dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:07:46.240505 11037 raft_consensus.cc:482] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:07:46.240926 11037 raft_consensus.cc:504] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8db306577b4fee8e86297833c402ee" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38639 } } peers { permanent_uuid: "6397ef0d7bfb492d93357d736f3a72dd" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 33979 } } peers { permanent_uuid: "cb6400349f4b436e8918ce7a3fe67507" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 41219 } }
I20260929 01:07:46.242735 11037 leader_election.cc:288] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b8db306577b4fee8e86297833c402ee (127.5.198.194:38639), 6397ef0d7bfb492d93357d736f3a72dd (127.5.198.193:33979)
I20260929 01:07:46.245538 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.243076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:38639 (local address 127.5.198.195:38651)
0929 01:07:46.243252 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:46.243274 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:46.243393 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:46.243970 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:46.243976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:46.244003 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:46.244661 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:07:46.244666 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:46.244687 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:07:46.244741 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:46.245106 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:46.245112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:46.245119 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 01:07:46.245233 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 01:07:46.245309 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 01:07:46.245605 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.243028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:38651 (local address 127.5.198.194:38639)
0929 01:07:46.243212 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:46.243218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:46.243377 (+ 159us) server_negotiation.cc:407] Connection header received
0929 01:07:46.243608 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:46.243616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:46.243679 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:46.243784 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:46.244906 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:46.244913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:46.244919 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:46.244938 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:46.244975 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:46.244994 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:46.245119 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:46.245238 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 01:07:46.245282 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 01:07:46.246102 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.243529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:39335 (local address 127.5.198.193:33979)
0929 01:07:46.243728 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:46.243733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:46.243975 (+ 242us) server_negotiation.cc:407] Connection header received
0929 01:07:46.244159 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:46.244165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:46.244248 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:46.244343 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:46.245408 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:46.245414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:46.245425 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:46.245446 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:46.245481 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:46.245496 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:46.245609 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:46.245775 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 01:07:46.245835 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 01:07:46.246572 10837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6504853e0b5f4e5784d91b033e98044e" candidate_uuid: "cb6400349f4b436e8918ce7a3fe67507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b8db306577b4fee8e86297833c402ee" is_pre_election: true
I20260929 01:07:46.246807 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.243201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:33979 (local address 127.5.198.195:39335)
0929 01:07:46.243859 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:46.243873 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:46.244028 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:46.244375 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:46.244377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:46.244393 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:46.245203 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:07:46.245206 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:46.245220 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:07:46.245257 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:46.245603 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:46.245608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:46.245613 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 01:07:46.246318 (+ 705us) client_negotiation.cc:240] Negotiation successful
0929 01:07:46.246370 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":414,"threads_started":1}
I20260929 01:07:46.247087 10837 raft_consensus.cc:2454] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6400349f4b436e8918ce7a3fe67507 in term 0.
I20260929 01:07:46.247807 10903 leader_election.cc:302] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b8db306577b4fee8e86297833c402ee, cb6400349f4b436e8918ce7a3fe67507; no voters:
I20260929 01:07:46.247745 10706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6504853e0b5f4e5784d91b033e98044e" candidate_uuid: "cb6400349f4b436e8918ce7a3fe67507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6397ef0d7bfb492d93357d736f3a72dd" is_pre_election: true
I20260929 01:07:46.248167 11037 raft_consensus.cc:2781] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:07:46.248227 10706 raft_consensus.cc:2454] T 6504853e0b5f4e5784d91b033e98044e P 6397ef0d7bfb492d93357d736f3a72dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6400349f4b436e8918ce7a3fe67507 in term 0.
I20260929 01:07:46.248320 11037 raft_consensus.cc:482] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:07:46.248493 11037 raft_consensus.cc:3037] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:46.250603 11037 raft_consensus.cc:504] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8db306577b4fee8e86297833c402ee" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38639 } } peers { permanent_uuid: "6397ef0d7bfb492d93357d736f3a72dd" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 33979 } } peers { permanent_uuid: "cb6400349f4b436e8918ce7a3fe67507" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 41219 } }
I20260929 01:07:46.251142 11037 leader_election.cc:288] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [CANDIDATE]: Term 1 election: Requested vote from peers 9b8db306577b4fee8e86297833c402ee (127.5.198.194:38639), 6397ef0d7bfb492d93357d736f3a72dd (127.5.198.193:33979)
I20260929 01:07:46.251449 10837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6504853e0b5f4e5784d91b033e98044e" candidate_uuid: "cb6400349f4b436e8918ce7a3fe67507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b8db306577b4fee8e86297833c402ee"
I20260929 01:07:46.251530 10706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6504853e0b5f4e5784d91b033e98044e" candidate_uuid: "cb6400349f4b436e8918ce7a3fe67507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6397ef0d7bfb492d93357d736f3a72dd"
I20260929 01:07:46.251633 10837 raft_consensus.cc:3037] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:46.251736 10706 raft_consensus.cc:3037] T 6504853e0b5f4e5784d91b033e98044e P 6397ef0d7bfb492d93357d736f3a72dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:46.253841 10706 raft_consensus.cc:2454] T 6504853e0b5f4e5784d91b033e98044e P 6397ef0d7bfb492d93357d736f3a72dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6400349f4b436e8918ce7a3fe67507 in term 1.
I20260929 01:07:46.253953 10837 raft_consensus.cc:2454] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6400349f4b436e8918ce7a3fe67507 in term 1.
I20260929 01:07:46.254216 10901 leader_election.cc:302] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6397ef0d7bfb492d93357d736f3a72dd, cb6400349f4b436e8918ce7a3fe67507; no voters:
I20260929 01:07:46.254526 11037 raft_consensus.cc:2781] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:46.254921 11037 raft_consensus.cc:686] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [term 1 LEADER]: Becoming Leader. State: Replica: cb6400349f4b436e8918ce7a3fe67507, State: Running, Role: LEADER
I20260929 01:07:46.255411 11037 consensus_queue.cc:237] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8db306577b4fee8e86297833c402ee" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38639 } } peers { permanent_uuid: "6397ef0d7bfb492d93357d736f3a72dd" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 33979 } } peers { permanent_uuid: "cb6400349f4b436e8918ce7a3fe67507" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 41219 } }
I20260929 01:07:46.259732 10564 catalog_manager.cc:5495] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb6400349f4b436e8918ce7a3fe67507 (127.5.198.195). New cstate: current_term: 1 leader_uuid: "cb6400349f4b436e8918ce7a3fe67507" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b8db306577b4fee8e86297833c402ee" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6397ef0d7bfb492d93357d736f3a72dd" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 33979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb6400349f4b436e8918ce7a3fe67507" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 41219 } health_report { overall_health: HEALTHY } } }
W20260929 01:07:46.272714 11013 tablet.cc:2369] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:07:46.275233 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.273468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.5.198.195:41219)
0929 01:07:46.273614 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:46.273618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:46.273746 (+ 128us) server_negotiation.cc:407] Connection header received
0929 01:07:46.273905 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:46.273908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:46.273984 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:46.274065 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:46.274824 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:46.274828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:46.274830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:46.274843 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:46.274869 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:46.274880 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:46.274977 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:46.275037 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 01:07:46.275075 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:07:46.282851 10837 raft_consensus.cc:1261] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee [term 1 FOLLOWER]: Refusing update from remote peer cb6400349f4b436e8918ce7a3fe67507: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:46.282892 10706 raft_consensus.cc:1261] T 6504853e0b5f4e5784d91b033e98044e P 6397ef0d7bfb492d93357d736f3a72dd [term 1 FOLLOWER]: Refusing update from remote peer cb6400349f4b436e8918ce7a3fe67507: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:07:46.283834 11040 consensus_queue.cc:1035] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6397ef0d7bfb492d93357d736f3a72dd" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 33979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:46.284169 11037 consensus_queue.cc:1035] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b8db306577b4fee8e86297833c402ee" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:46.295173 11043 mvcc.cc:205] Tried to move back new op lower bound from 7334478095485231104 to 7334478095386759168. Current Snapshot: MvccSnapshot[applied={T|T < 7334478095485231104}]
I20260929 01:07:46.295768 11044 mvcc.cc:205] Tried to move back new op lower bound from 7334478095485231104 to 7334478095386759168. Current Snapshot: MvccSnapshot[applied={T|T < 7334478095485231104}]
I20260929 01:07:46.300860 11045 mvcc.cc:205] Tried to move back new op lower bound from 7334478095485231104 to 7334478095386759168. Current Snapshot: MvccSnapshot[applied={T|T < 7334478095485231104}]
I20260929 01:07:46.310058 10564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:07:46.310324 10564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:07:46.313700 10564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4cdeb31c7e64deb979c721427708e58: Sending DeleteTablet for 3 replicas of tablet 6504853e0b5f4e5784d91b033e98044e
I20260929 01:07:46.314733 10817 tablet_service.cc:1514] Processing DeleteTablet for tablet 6504853e0b5f4e5784d91b033e98044e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:46 UTC) from {username='slave'} at 127.0.0.1:53428
I20260929 01:07:46.314877 10686 tablet_service.cc:1514] Processing DeleteTablet for tablet 6504853e0b5f4e5784d91b033e98044e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:46 UTC) from {username='slave'} at 127.0.0.1:41100
I20260929 01:07:46.315135 10947 tablet_service.cc:1514] Processing DeleteTablet for tablet 6504853e0b5f4e5784d91b033e98044e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:07:46 UTC) from {username='slave'} at 127.0.0.1:48206
I20260929 01:07:46.316658 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 10624
I20260929 01:07:46.327847 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.317928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.5.198.193:33979)
0929 01:07:46.318108 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:46.318114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:46.323480 (+ 5366us) server_negotiation.cc:407] Connection header received
0929 01:07:46.324527 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:46.324536 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:46.324663 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:46.324754 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:46.326639 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:46.326649 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:46.326657 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:46.326683 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:46.326730 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:46.326751 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:46.326866 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:46.327330 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 01:07:46.327407 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 01:07:46.329185 10746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:46.331398 11052 tablet_replica.cc:330] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee: stopping tablet replica
I20260929 01:07:46.332103 11052 raft_consensus.cc:2229] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:07:46.332834 11052 raft_consensus.cc:2258] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:46.334681 11054 tablet_replica.cc:330] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507: stopping tablet replica
I20260929 01:07:46.335216 11054 raft_consensus.cc:2229] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:07:46.335960 11054 raft_consensus.cc:2258] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:46.336046 11052 ts_tablet_manager.cc:1905] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:46.338840 11054 ts_tablet_manager.cc:1905] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:46.343086 11052 ts_tablet_manager.cc:1918] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:07:46.343253 11052 log.cc:1196] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/6504853e0b5f4e5784d91b033e98044e
I20260929 01:07:46.343772 11052 ts_tablet_manager.cc:1939] T 6504853e0b5f4e5784d91b033e98044e P 9b8db306577b4fee8e86297833c402ee: Deleting consensus metadata
I20260929 01:07:46.345623 10552 catalog_manager.cc:4841] TS 9b8db306577b4fee8e86297833c402ee (127.5.198.194:38639): tablet 6504853e0b5f4e5784d91b033e98044e (table test-table [id=acee23f8c6864777abcce759e24668a2]) successfully deleted
I20260929 01:07:46.346567 11054 ts_tablet_manager.cc:1918] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:07:46.346740 11054 log.cc:1196] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/6504853e0b5f4e5784d91b033e98044e
I20260929 01:07:46.347218 11054 ts_tablet_manager.cc:1939] T 6504853e0b5f4e5784d91b033e98044e P cb6400349f4b436e8918ce7a3fe67507: Deleting consensus metadata
I20260929 01:07:46.348603 10551 catalog_manager.cc:4841] TS cb6400349f4b436e8918ce7a3fe67507 (127.5.198.195:41219): tablet 6504853e0b5f4e5784d91b033e98044e (table test-table [id=acee23f8c6864777abcce759e24668a2]) successfully deleted
I20260929 01:07:46.447929 11053 tablet_replica.cc:330] T 6504853e0b5f4e5784d91b033e98044e P 6397ef0d7bfb492d93357d736f3a72dd: stopping tablet replica
I20260929 01:07:46.448264 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 10624
I20260929 01:07:46.465513 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 10755
W20260929 01:07:46.465492 10550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.198.193:33979 (error 108)
I20260929 01:07:46.467463 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.466978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:33979 (local address 127.0.0.1:41118)
0929 01:07:46.467141 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:46.467262 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 01:07:46.467835 10550 catalog_manager.cc:4584] TS 6397ef0d7bfb492d93357d736f3a72dd (127.5.198.193:33979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6504853e0b5f4e5784d91b033e98044e: Network error: Client connection negotiation failed: client connection to 127.5.198.193:33979: connect: Connection refused (error 111)
I20260929 01:07:46.468858 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.466597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.5.198.194:38639)
0929 01:07:46.466766 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:46.466773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:46.466794 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:07:46.466856 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:46.466861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:46.466964 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:46.467069 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:46.468202 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:46.468209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:46.468214 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:46.468234 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:46.468272 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:46.468292 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:46.468432 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:46.468512 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 01:07:46.468571 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:07:46.469538 10877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:46.521534 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.520959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:33979 (local address 127.0.0.1:41134)
0929 01:07:46.521228 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:46.521343 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 01:07:46.522652 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.522263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:33979 (local address 127.0.0.1:41142)
0929 01:07:46.522421 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:46.522503 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 01:07:46.557549 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 10755
I20260929 01:07:46.579799 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 10885
I20260929 01:07:46.583468 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.582941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:33979 (local address 127.0.0.1:41146)
0929 01:07:46.583162 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:46.583249 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 01:07:46.583933 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.580623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.5.198.195:41219)
0929 01:07:46.580834 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:46.580840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:46.580864 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:07:46.581034 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:46.581041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:46.581107 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:46.581220 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:46.583158 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:46.583164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:46.583170 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:46.583191 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:46.583229 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:46.583247 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:46.583433 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:46.583589 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 01:07:46.583648 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:07:46.584717 11007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:46.585476 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.584967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:33979 (local address 127.0.0.1:41148)
0929 01:07:46.585196 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:46.585291 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 01:07:46.666119 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.665703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:33979 (local address 127.0.0.1:41152)
0929 01:07:46.665897 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:46.665990 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 01:07:46.667068 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:46.666709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:33979 (local address 127.0.0.1:41166)
0929 01:07:46.666888 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:46.666972 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 01:07:46.701579 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 10885
I20260929 01:07:46.721194 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 10534
I20260929 01:07:46.722309 10595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:46.825832 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 10534
2026-09-29T01:07:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2633 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 01:07:46.866003 5915 mini_oidc.cc:152] Starting JWKS server
I20260929 01:07:46.868405 5915 webserver.cc:460] Webserver started at https://127.0.0.1:38897/ using document root <none> and password file <none>
I20260929 01:07:46.868749 5915 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 01:07:46.869369 5915 webserver.cc:460] Webserver started at http://127.0.0.1:45699/ using document root <none> and password file <none>
2026-09-29T01:07:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:07:46Z Disabled control of system clock
I20260929 01:07:46.894665 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:40899
--webserver_interface=127.5.198.254
--webserver_port=0
--builtin_ntp_servers=127.5.198.212:44889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:40899
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38897/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/testchainca.cert with env {}
W20260929 01:07:47.204530 11066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:47.204850 11066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:47.204943 11066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:47.212107 11066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:47.212205 11066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:47.212241 11066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:47.212296 11066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:47.212342 11066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 01:07:47.212393 11066 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 01:07:47.212432 11066 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38897/jwks.json
I20260929 01:07:47.222024 11066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:44889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:40899
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:40899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38897/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:47.223151 11066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:47.224324 11066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:47.235833 11071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:47.235895 11074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:47.235873 11072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:47.236653 11066 server_base.cc:1034] running on GCE node
I20260929 01:07:47.237419 11066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:47.238725 11066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:47.239941 11066 hybrid_clock.cc:648] HybridClock initialized: now 1790644067239896 us; error 47 us; skew 500 ppm
I20260929 01:07:47.240276 11066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:07:47.247694 11066 webserver.cc:460] Webserver started at http://127.5.198.254:35233/ using document root <none> and password file <none>
I20260929 01:07:47.248359 11066 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:47.248535 11066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:47.248840 11066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:47.251552 11066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "b17e06ad8fa6429ab20b1f637fa3f40d"
format_stamp: "Formatted at 2026-09-29 01:07:47 on dist-test-slave-41cl"
server_key: "a68954074ce6882857d6b294fc1b77c2"
server_key_iv: "19be8e578cf6e6cdc354711eacf6861b"
server_key_version: "encryptionkey@0"
I20260929 01:07:47.252310 11066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "b17e06ad8fa6429ab20b1f637fa3f40d"
format_stamp: "Formatted at 2026-09-29 01:07:47 on dist-test-slave-41cl"
server_key: "a68954074ce6882857d6b294fc1b77c2"
server_key_iv: "19be8e578cf6e6cdc354711eacf6861b"
server_key_version: "encryptionkey@0"
I20260929 01:07:47.257143 11066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:07:47.260294 11080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:47.261575 11066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:07:47.261747 11066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "b17e06ad8fa6429ab20b1f637fa3f40d"
format_stamp: "Formatted at 2026-09-29 01:07:47 on dist-test-slave-41cl"
server_key: "a68954074ce6882857d6b294fc1b77c2"
server_key_iv: "19be8e578cf6e6cdc354711eacf6861b"
server_key_version: "encryptionkey@0"
I20260929 01:07:47.261950 11066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:47.287366 11066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:47.288162 11066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:47.288545 11066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:47.307307 11066 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:40899
I20260929 01:07:47.307327 11131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:40899 every 8 connection(s)
I20260929 01:07:47.308610 11066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:07:47.311393 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 11066
I20260929 01:07:47.311615 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:07:47.312364 5915 external_mini_cluster.cc:1428] Setting key 8ca37e2d66cca2027dfc98bed6315de8
I20260929 01:07:47.312484 11132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:47.325479 11132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d: Bootstrap starting.
I20260929 01:07:47.325496 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:47.314194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.5.198.254:40899)
0929 01:07:47.315161 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:47.315171 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:47.315211 (+ 40us) server_negotiation.cc:407] Connection header received
0929 01:07:47.316934 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:47.316954 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:47.317387 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:47.317693 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:47.318301 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:47.319577 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:47.321313 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:47.321913 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:47.322066 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:47.323330 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:47.323337 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:47.323350 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:47.323873 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:47.323942 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:47.323969 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:47.324055 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:47.324262 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 01:07:47.324610 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":366,"threads_started":1}
I20260929 01:07:47.328905 11132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:47.329197 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:07:47.329907 11132 log.cc:824] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:47.332662 11132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d: No bootstrap required, opened a new log
I20260929 01:07:47.340088 11132 raft_consensus.cc:348] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17e06ad8fa6429ab20b1f637fa3f40d" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40899 } }
I20260929 01:07:47.340410 11132 raft_consensus.cc:374] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:47.340544 11132 raft_consensus.cc:729] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b17e06ad8fa6429ab20b1f637fa3f40d, State: Initialized, Role: FOLLOWER
I20260929 01:07:47.341115 11132 consensus_queue.cc:260] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17e06ad8fa6429ab20b1f637fa3f40d" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40899 } }
I20260929 01:07:47.341331 11132 raft_consensus.cc:388] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:07:47.341462 11132 raft_consensus.cc:482] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:07:47.341629 11132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:47.343591 11132 raft_consensus.cc:504] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17e06ad8fa6429ab20b1f637fa3f40d" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40899 } }
I20260929 01:07:47.344094 11132 leader_election.cc:302] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b17e06ad8fa6429ab20b1f637fa3f40d; no voters:
I20260929 01:07:47.344738 11132 leader_election.cc:288] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:07:47.344873 11137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:47.345554 11137 raft_consensus.cc:686] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [term 1 LEADER]: Becoming Leader. State: Replica: b17e06ad8fa6429ab20b1f637fa3f40d, State: Running, Role: LEADER
I20260929 01:07:47.346045 11137 consensus_queue.cc:237] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17e06ad8fa6429ab20b1f637fa3f40d" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40899 } }
I20260929 01:07:47.346662 11132 sys_catalog.cc:564] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:47.348869 11139 sys_catalog.cc:455] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b17e06ad8fa6429ab20b1f637fa3f40d. Latest consensus state: current_term: 1 leader_uuid: "b17e06ad8fa6429ab20b1f637fa3f40d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17e06ad8fa6429ab20b1f637fa3f40d" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40899 } } }
I20260929 01:07:47.348815 11138 sys_catalog.cc:455] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b17e06ad8fa6429ab20b1f637fa3f40d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17e06ad8fa6429ab20b1f637fa3f40d" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40899 } } }
I20260929 01:07:47.349225 11139 sys_catalog.cc:458] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:47.349226 11138 sys_catalog.cc:458] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:47.351456 11144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:07:47.357805 11144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:07:47.366081 11144 catalog_manager.cc:1347] Generated new cluster ID: a587d27fc7244bf49d0b5f92bd9b95c0
I20260929 01:07:47.366195 11144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:07:47.379503 11144 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:07:47.381263 11144 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:07:47.402176 11144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b17e06ad8fa6429ab20b1f637fa3f40d: Generated new TSK 0
I20260929 01:07:47.403151 11144 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 01:07:47.439618 11162 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 01:07:47.439796 11162 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 01:07:47.440791 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:47.419563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53296 (local address 127.5.198.254:40899)
0929 01:07:47.419781 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:47.419790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:47.420250 (+ 460us) server_negotiation.cc:407] Connection header received
0929 01:07:47.420586 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:47.420590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:47.420653 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:47.420760 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0929 01:07:47.421360 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:47.422049 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:47.424105 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:47.424559 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:47.424656 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:47.425276 (+ 620us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 01:07:47.440232 (+ 14956us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38897/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 01:07:47.440564 (+ 332us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53296: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38897/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 01:07:47.440927 11135 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53296: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38897/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 01:07:47.442698 11161 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:07:47.419501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:40899 (local address 127.0.0.1:53296)
0929 01:07:47.420147 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:47.420166 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:47.420337 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:47.420823 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:47.420829 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:47.420842 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 01:07:47.421219 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:47.421232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:47.422187 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:47.422193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:47.423973 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:47.423988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:47.424730 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:47.424737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:47.424917 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:47.425001 (+ 84us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 01:07:47.440537 (+ 15536us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38897/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 01:07:47.440881 (+ 344us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.198.254:40899: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38897/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":320,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.198.254:40899: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38897/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 01:07:47.444319 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 11066
I20260929 01:07:47.445513 11127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:47.537618 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 11066
2026-09-29T01:07:47Z chronyd exiting
I20260929 01:07:47.587821 5915 test_util.cc:175] -----------------------------------------------
I20260929 01:07:47.587976 5915 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790644023292677-5915-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (726 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 01:07:47.591948 5915 mini_oidc.cc:152] Starting JWKS server
I20260929 01:07:47.593941 5915 webserver.cc:460] Webserver started at https://127.0.0.1:45043/ using document root <none> and password file <none>
I20260929 01:07:47.594172 5915 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 01:07:47.594939 5915 webserver.cc:460] Webserver started at http://127.0.0.1:34097/ using document root <none> and password file <none>
2026-09-29T01:07:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:07:47Z Disabled control of system clock
I20260929 01:07:47.621254 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:34023
--webserver_interface=127.5.198.254
--webserver_port=0
--builtin_ntp_servers=127.5.198.212:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:34023
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45043/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/testchainca.cert with env {}
W20260929 01:07:47.932615 11169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:47.932943 11169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:47.933110 11169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:47.940178 11169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:47.940325 11169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:47.940418 11169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:47.940537 11169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:47.940605 11169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 01:07:47.940692 11169 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 01:07:47.940773 11169 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45043/jwks.json
I20260929 01:07:47.950407 11169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:34023
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:34023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45043/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:47.951682 11169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:47.952986 11169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:47.964720 11175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:47.964614 11177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:47.964666 11174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:47.965206 11169 server_base.cc:1034] running on GCE node
I20260929 01:07:47.966259 11169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:47.967633 11169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:47.968842 11169 hybrid_clock.cc:648] HybridClock initialized: now 1790644067968798 us; error 38 us; skew 500 ppm
I20260929 01:07:47.969218 11169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:07:47.971529 11169 webserver.cc:460] Webserver started at http://127.5.198.254:36437/ using document root <none> and password file <none>
I20260929 01:07:47.972199 11169 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:47.972311 11169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:47.972688 11169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:47.976953 11169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "074a3fb976c447e0bd41c91d698e45bb"
format_stamp: "Formatted at 2026-09-29 01:07:47 on dist-test-slave-41cl"
server_key: "68a9c670d0771090de89b73dbef1d20f"
server_key_iv: "d19dba7bd798b4a8c8650dec486eb509"
server_key_version: "encryptionkey@0"
I20260929 01:07:47.977708 11169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "074a3fb976c447e0bd41c91d698e45bb"
format_stamp: "Formatted at 2026-09-29 01:07:47 on dist-test-slave-41cl"
server_key: "68a9c670d0771090de89b73dbef1d20f"
server_key_iv: "d19dba7bd798b4a8c8650dec486eb509"
server_key_version: "encryptionkey@0"
I20260929 01:07:47.982514 11169 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:07:47.985877 11183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:47.987170 11169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:07:47.987345 11169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "074a3fb976c447e0bd41c91d698e45bb"
format_stamp: "Formatted at 2026-09-29 01:07:47 on dist-test-slave-41cl"
server_key: "68a9c670d0771090de89b73dbef1d20f"
server_key_iv: "d19dba7bd798b4a8c8650dec486eb509"
server_key_version: "encryptionkey@0"
I20260929 01:07:47.987555 11169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:48.006448 11169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:48.007277 11169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:48.007594 11169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:48.025648 11169 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:34023
I20260929 01:07:48.025669 11234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:34023 every 8 connection(s)
I20260929 01:07:48.026959 11169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:07:48.027740 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 11169
I20260929 01:07:48.028005 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:07:48.028707 5915 external_mini_cluster.cc:1428] Setting key 4283ec5afa5d3abaf4a39d1794dbf825
I20260929 01:07:48.031286 11235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:48.041255 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:48.030509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.5.198.254:34023)
0929 01:07:48.031504 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:48.031514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:48.031552 (+ 38us) server_negotiation.cc:407] Connection header received
0929 01:07:48.033100 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:48.033119 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:48.033543 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:48.033934 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:48.034475 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:48.035757 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:48.037075 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:48.037699 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:48.037900 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:48.039008 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:48.039016 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:48.039029 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:48.039556 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:48.039627 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:48.039649 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:48.039772 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:48.039982 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 01:07:48.040256 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":362,"threads_started":1}
I20260929 01:07:48.041615 11235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb: Bootstrap starting.
I20260929 01:07:48.045217 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:07:48.045248 11235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:48.046183 11235 log.cc:824] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:48.049154 11235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb: No bootstrap required, opened a new log
I20260929 01:07:48.056279 11235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a3fb976c447e0bd41c91d698e45bb" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34023 } }
I20260929 01:07:48.056711 11235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:48.056818 11235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a3fb976c447e0bd41c91d698e45bb, State: Initialized, Role: FOLLOWER
I20260929 01:07:48.057367 11235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a3fb976c447e0bd41c91d698e45bb" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34023 } }
I20260929 01:07:48.057638 11235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:07:48.057765 11235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:07:48.057952 11235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:48.059939 11235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a3fb976c447e0bd41c91d698e45bb" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34023 } }
I20260929 01:07:48.060398 11235 leader_election.cc:302] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 074a3fb976c447e0bd41c91d698e45bb; no voters:
I20260929 01:07:48.061014 11235 leader_election.cc:288] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:07:48.061203 11240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:48.062193 11240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [term 1 LEADER]: Becoming Leader. State: Replica: 074a3fb976c447e0bd41c91d698e45bb, State: Running, Role: LEADER
I20260929 01:07:48.062708 11240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a3fb976c447e0bd41c91d698e45bb" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34023 } }
I20260929 01:07:48.062965 11235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:48.065452 11241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "074a3fb976c447e0bd41c91d698e45bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a3fb976c447e0bd41c91d698e45bb" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34023 } } }
I20260929 01:07:48.065397 11242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 074a3fb976c447e0bd41c91d698e45bb. Latest consensus state: current_term: 1 leader_uuid: "074a3fb976c447e0bd41c91d698e45bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a3fb976c447e0bd41c91d698e45bb" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34023 } } }
I20260929 01:07:48.065861 11241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:48.065861 11242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:48.066790 11249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:07:48.071370 11249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:07:48.079084 11249 catalog_manager.cc:1347] Generated new cluster ID: 6d4731ab09bc4fd89490172fca756dee
I20260929 01:07:48.079208 11249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:07:48.102303 11249 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:07:48.103515 11249 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:07:48.117209 11249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 074a3fb976c447e0bd41c91d698e45bb: Generated new TSK 0
I20260929 01:07:48.118026 11249 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 01:07:48.157183 11265 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 01:07:48.157318 11265 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 01:07:48.158497 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:48.131738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41442 (local address 127.5.198.254:34023)
0929 01:07:48.131911 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:48.131918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:48.132270 (+ 352us) server_negotiation.cc:407] Connection header received
0929 01:07:48.132537 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:48.132540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:48.132612 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:48.132755 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0929 01:07:48.133217 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:48.133901 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:48.135230 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:48.135681 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:48.135811 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:48.136261 (+ 450us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 01:07:48.157674 (+ 21413us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 01:07:48.158111 (+ 437us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41442: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 01:07:48.158617 11264 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:07:48.131656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:34023 (local address 127.0.0.1:41442)
0929 01:07:48.132171 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:48.132189 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:48.132344 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:48.132770 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:48.132775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:48.132787 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0929 01:07:48.133057 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:48.133066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:48.134050 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:48.134062 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:48.135077 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:48.135087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:48.135832 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:48.135838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:48.135990 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:48.136020 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 01:07:48.157928 (+ 21908us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 01:07:48.158190 (+ 262us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.198.254:34023: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":293,"threads_started":1}
W20260929 01:07:48.158905 11238 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41442: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260929 01:07:48.159786 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 11169
I20260929 01:07:48.161046 11230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:48.253679 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 11169
2026-09-29T01:07:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (717 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 01:07:48.310423 5915 mini_oidc.cc:152] Starting JWKS server
I20260929 01:07:48.312122 5915 webserver.cc:460] Webserver started at https://127.0.0.1:33319/ using document root <none> and password file <none>
I20260929 01:07:48.312326 5915 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 01:07:48.312903 5915 webserver.cc:460] Webserver started at http://127.0.0.1:44751/ using document root <none> and password file <none>
2026-09-29T01:07:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:07:48Z Disabled control of system clock
I20260929 01:07:48.338157 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:41009
--webserver_interface=127.5.198.254
--webserver_port=0
--builtin_ntp_servers=127.5.198.212:40539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:41009
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33319/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 01:07:48.647594 11272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:48.647857 11272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:48.648059 11272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:48.655031 11272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:48.655125 11272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:48.655218 11272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:48.655295 11272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:48.655356 11272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 01:07:48.655460 11272 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 01:07:48.655535 11272 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33319/jwks.json
I20260929 01:07:48.664284 11272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:41009
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:41009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33319/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:48.665530 11272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:48.667263 11272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:48.678804 11277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:48.678819 11278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:48.678845 11280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:48.679630 11272 server_base.cc:1034] running on GCE node
I20260929 01:07:48.680392 11272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:48.681775 11272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:48.683010 11272 hybrid_clock.cc:648] HybridClock initialized: now 1790644068682962 us; error 32 us; skew 500 ppm
I20260929 01:07:48.683390 11272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:07:48.685676 11272 webserver.cc:460] Webserver started at http://127.5.198.254:41827/ using document root <none> and password file <none>
I20260929 01:07:48.686391 11272 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:48.686506 11272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:48.686841 11272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:48.689734 11272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "a49389fceaea4e05974626e7e7ece06a"
format_stamp: "Formatted at 2026-09-29 01:07:48 on dist-test-slave-41cl"
server_key: "5ec1d25945055eaaa510193f8a8063a3"
server_key_iv: "3937c9adc7aae480a40b35a0a3858575"
server_key_version: "encryptionkey@0"
I20260929 01:07:48.690516 11272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "a49389fceaea4e05974626e7e7ece06a"
format_stamp: "Formatted at 2026-09-29 01:07:48 on dist-test-slave-41cl"
server_key: "5ec1d25945055eaaa510193f8a8063a3"
server_key_iv: "3937c9adc7aae480a40b35a0a3858575"
server_key_version: "encryptionkey@0"
I20260929 01:07:48.695590 11272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260929 01:07:48.699009 11286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:48.700320 11272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:07:48.700562 11272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "a49389fceaea4e05974626e7e7ece06a"
format_stamp: "Formatted at 2026-09-29 01:07:48 on dist-test-slave-41cl"
server_key: "5ec1d25945055eaaa510193f8a8063a3"
server_key_iv: "3937c9adc7aae480a40b35a0a3858575"
server_key_version: "encryptionkey@0"
I20260929 01:07:48.700786 11272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:48.730479 11272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:48.731391 11272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:48.731709 11272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:48.749864 11272 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:41009
I20260929 01:07:48.749884 11337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:41009 every 8 connection(s)
I20260929 01:07:48.751173 11272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:07:48.754732 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 11272
I20260929 01:07:48.754938 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:07:48.755030 11338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:48.755733 5915 external_mini_cluster.cc:1428] Setting key 74ebf8736f2f74808f3a3315a0aa4989
I20260929 01:07:48.768317 11338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a: Bootstrap starting.
I20260929 01:07:48.769801 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:48.757696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39962 (local address 127.5.198.254:41009)
0929 01:07:48.758703 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:48.758713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:48.758752 (+ 39us) server_negotiation.cc:407] Connection header received
0929 01:07:48.761090 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:48.761111 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:48.761547 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:48.761864 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:48.762404 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:48.763662 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:48.765565 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:48.766273 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:48.766398 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:48.767564 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:48.767571 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:48.767584 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:48.768096 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:48.768166 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:48.768185 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:48.768273 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:48.768578 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 01:07:48.768898 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":395,"threads_started":1}
I20260929 01:07:48.771411 11338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:48.772547 11338 log.cc:824] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:48.773413 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:07:48.775198 11338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a: No bootstrap required, opened a new log
I20260929 01:07:48.782444 11338 raft_consensus.cc:348] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49389fceaea4e05974626e7e7ece06a" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 41009 } }
I20260929 01:07:48.782779 11338 raft_consensus.cc:374] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:48.782891 11338 raft_consensus.cc:729] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49389fceaea4e05974626e7e7ece06a, State: Initialized, Role: FOLLOWER
I20260929 01:07:48.783411 11338 consensus_queue.cc:260] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49389fceaea4e05974626e7e7ece06a" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 41009 } }
I20260929 01:07:48.783653 11338 raft_consensus.cc:388] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:07:48.783792 11338 raft_consensus.cc:482] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:07:48.783964 11338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:48.786315 11338 raft_consensus.cc:504] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49389fceaea4e05974626e7e7ece06a" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 41009 } }
I20260929 01:07:48.786788 11338 leader_election.cc:302] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a49389fceaea4e05974626e7e7ece06a; no voters:
I20260929 01:07:48.787338 11338 leader_election.cc:288] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:07:48.787479 11343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:48.788276 11343 raft_consensus.cc:686] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [term 1 LEADER]: Becoming Leader. State: Replica: a49389fceaea4e05974626e7e7ece06a, State: Running, Role: LEADER
I20260929 01:07:48.788794 11343 consensus_queue.cc:237] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49389fceaea4e05974626e7e7ece06a" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 41009 } }
I20260929 01:07:48.789310 11338 sys_catalog.cc:564] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:48.791338 11344 sys_catalog.cc:455] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a49389fceaea4e05974626e7e7ece06a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49389fceaea4e05974626e7e7ece06a" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 41009 } } }
I20260929 01:07:48.791668 11344 sys_catalog.cc:458] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:48.792896 11345 sys_catalog.cc:455] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a49389fceaea4e05974626e7e7ece06a. Latest consensus state: current_term: 1 leader_uuid: "a49389fceaea4e05974626e7e7ece06a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49389fceaea4e05974626e7e7ece06a" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 41009 } } }
I20260929 01:07:48.793268 11345 sys_catalog.cc:458] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:48.794080 11349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:07:48.798947 11349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:07:48.807428 11349 catalog_manager.cc:1347] Generated new cluster ID: 1042a38596984ed8b5d864e202e15afb
I20260929 01:07:48.807526 11349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:07:48.850449 11349 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:07:48.851715 11349 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:07:48.857442 11349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a49389fceaea4e05974626e7e7ece06a: Generated new TSK 0
I20260929 01:07:48.858343 11349 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:07:48.894183 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:48.885273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39966 (local address 127.5.198.254:41009)
0929 01:07:48.885498 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:48.885505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:48.885983 (+ 478us) server_negotiation.cc:407] Connection header received
0929 01:07:48.886199 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:48.886205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:48.886297 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:48.886461 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0929 01:07:48.886978 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:48.888114 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:48.889728 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:48.890481 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:48.890609 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:48.891220 (+ 611us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 01:07:48.893568 (+ 2348us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33319/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 01:07:48.893886 (+ 318us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39966: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33319/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 01:07:48.894308 11341 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39966: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33319/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 01:07:48.896103 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:07:48.885111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:41009 (local address 127.0.0.1:39966)
0929 01:07:48.885879 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:48.885896 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:48.886023 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:48.886493 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:48.886497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:48.886507 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 01:07:48.886821 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:48.886831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:48.888273 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:48.888278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:48.889561 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:48.889573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:48.890676 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:48.890681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:48.890888 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:48.890948 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 01:07:48.893757 (+ 2809us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33319/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 01:07:48.895694 (+ 1937us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.198.254:41009: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33319/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":1649,"thread_start_us":390,"threads_started":1}
I20260929 01:07:48.897033 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 11272
I20260929 01:07:48.898065 11333 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 01:07:48.906278 11368 webserver.cc:533] Webserver: sslize failed:
E20260929 01:07:48.906404 11368 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 01:07:48.993026 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 11272
2026-09-29T01:07:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (738 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T01:07:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:07:49Z Disabled control of system clock
I20260929 01:07:49.073032 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:38943
--webserver_interface=127.5.198.254
--webserver_port=0
--builtin_ntp_servers=127.5.198.212:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:38943
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/testchainca.cert with env {}
W20260929 01:07:49.383548 11373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:49.383826 11373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:49.383910 11373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:49.390962 11373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:49.391053 11373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:49.391090 11373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:49.391136 11373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:49.391179 11373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 01:07:49.391240 11373 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 01:07:49.391279 11373 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 01:07:49.401154 11373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:38943
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:38943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:49.402602 11373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:49.403746 11373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:49.414958 11378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:49.414882 11379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:49.414961 11381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:49.415768 11373 server_base.cc:1034] running on GCE node
I20260929 01:07:49.416575 11373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:49.417883 11373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:49.419113 11373 hybrid_clock.cc:648] HybridClock initialized: now 1790644069419071 us; error 54 us; skew 500 ppm
I20260929 01:07:49.419440 11373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:07:49.421593 11373 webserver.cc:460] Webserver started at http://127.5.198.254:36099/ using document root <none> and password file <none>
I20260929 01:07:49.422281 11373 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:49.422410 11373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:49.422716 11373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:49.425513 11373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "bf36f7fc98344eff8a168677a3aff80f"
format_stamp: "Formatted at 2026-09-29 01:07:49 on dist-test-slave-41cl"
server_key: "295fae2fb0a6ee164a921a73f5d8aeb5"
server_key_iv: "52f1a11952243c94a56e139c54dac103"
server_key_version: "encryptionkey@0"
I20260929 01:07:49.426263 11373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "bf36f7fc98344eff8a168677a3aff80f"
format_stamp: "Formatted at 2026-09-29 01:07:49 on dist-test-slave-41cl"
server_key: "295fae2fb0a6ee164a921a73f5d8aeb5"
server_key_iv: "52f1a11952243c94a56e139c54dac103"
server_key_version: "encryptionkey@0"
I20260929 01:07:49.431002 11373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:07:49.434211 11387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:49.435539 11373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:07:49.435755 11373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "bf36f7fc98344eff8a168677a3aff80f"
format_stamp: "Formatted at 2026-09-29 01:07:49 on dist-test-slave-41cl"
server_key: "295fae2fb0a6ee164a921a73f5d8aeb5"
server_key_iv: "52f1a11952243c94a56e139c54dac103"
server_key_version: "encryptionkey@0"
I20260929 01:07:49.435988 11373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:49.450120 11373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:49.450892 11373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:49.451190 11373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:49.469871 11373 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:38943
I20260929 01:07:49.469873 11438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:38943 every 8 connection(s)
I20260929 01:07:49.471223 11373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:07:49.474699 11439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:49.479907 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 11373
I20260929 01:07:49.480110 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:07:49.480921 5915 external_mini_cluster.cc:1428] Setting key 037584059a8cc43c60b83059dff2849f
I20260929 01:07:49.485572 11439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f: Bootstrap starting.
I20260929 01:07:49.488348 11439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:49.489315 11439 log.cc:824] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:49.492271 11439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f: No bootstrap required, opened a new log
I20260929 01:07:49.493691 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:49.482759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51468 (local address 127.5.198.254:38943)
0929 01:07:49.483710 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:49.483720 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:49.483763 (+ 43us) server_negotiation.cc:407] Connection header received
0929 01:07:49.485379 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:49.485401 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:49.485863 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:49.486221 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:49.486741 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:49.487940 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:49.489256 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:49.490196 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:49.490402 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:49.491498 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:49.491505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:49.491518 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:07:49.492019 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:49.492088 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:07:49.492107 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:49.492195 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:49.492514 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 01:07:49.492836 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":337,"threads_started":1}
I20260929 01:07:49.497249 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:07:49.500790 11439 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf36f7fc98344eff8a168677a3aff80f" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38943 } }
I20260929 01:07:49.501109 11439 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:49.501205 11439 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf36f7fc98344eff8a168677a3aff80f, State: Initialized, Role: FOLLOWER
I20260929 01:07:49.501711 11439 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf36f7fc98344eff8a168677a3aff80f" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38943 } }
I20260929 01:07:49.501914 11439 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:07:49.502069 11439 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:07:49.502207 11439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:49.504132 11439 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf36f7fc98344eff8a168677a3aff80f" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38943 } }
I20260929 01:07:49.504611 11439 leader_election.cc:302] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf36f7fc98344eff8a168677a3aff80f; no voters:
I20260929 01:07:49.505159 11439 leader_election.cc:288] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:07:49.505327 11444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:49.506042 11444 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [term 1 LEADER]: Becoming Leader. State: Replica: bf36f7fc98344eff8a168677a3aff80f, State: Running, Role: LEADER
I20260929 01:07:49.506582 11444 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf36f7fc98344eff8a168677a3aff80f" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38943 } }
I20260929 01:07:49.507165 11439 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:49.509310 11446 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf36f7fc98344eff8a168677a3aff80f. Latest consensus state: current_term: 1 leader_uuid: "bf36f7fc98344eff8a168677a3aff80f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf36f7fc98344eff8a168677a3aff80f" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38943 } } }
I20260929 01:07:49.509654 11446 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:49.510519 11452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:07:49.510401 11445 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf36f7fc98344eff8a168677a3aff80f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf36f7fc98344eff8a168677a3aff80f" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38943 } } }
I20260929 01:07:49.510772 11445 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:49.515784 11452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:07:49.524183 11452 catalog_manager.cc:1347] Generated new cluster ID: 8487add4ec0942adbd6a8ccca3eaa75e
I20260929 01:07:49.524339 11452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:07:49.552529 11452 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:07:49.553812 11452 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:07:49.562885 11452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf36f7fc98344eff8a168677a3aff80f: Generated new TSK 0
I20260929 01:07:49.563690 11452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:07:49.645013 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:49.581838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.5.198.254:38943)
0929 01:07:49.582106 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:49.582113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:49.582509 (+ 396us) server_negotiation.cc:407] Connection header received
0929 01:07:49.582712 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:49.582716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:49.582784 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:49.582936 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0929 01:07:49.583550 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:49.584689 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:49.586806 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:49.587298 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:49.587405 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:49.588141 (+ 736us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 01:07:49.644348 (+ 56207us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 01:07:49.644751 (+ 403us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51484: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":103}
W20260929 01:07:49.645172 11442 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51484: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 01:07:49.645725 11468 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:07:49.581738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38943 (local address 127.0.0.1:51484)
0929 01:07:49.582345 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:49.582365 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:49.582521 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:49.582980 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:49.582986 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:49.582999 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 01:07:49.583319 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:49.583342 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:49.584861 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:49.584868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:49.586635 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:49.586652 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:49.587488 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:49.587495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:49.587711 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:49.587758 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 01:07:49.644678 (+ 56920us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 01:07:49.644992 (+ 314us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.198.254:38943: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":292,"threads_started":1}
I20260929 01:07:49.647269 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 11373
I20260929 01:07:49.648423 11434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:49.744733 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 11373
2026-09-29T01:07:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (731 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (408 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (396 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:07:50 dist-test-slave-41cl krb5kdc[11483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:07:50 dist-test-slave-41cl krb5kdc[11483](info): set up 2 sockets
Sep 29 01:07:50 dist-test-slave-41cl krb5kdc[11483](info): commencing operation
krb5kdc: starting...
W20260929 01:07:51.637194 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:07:51 dist-test-slave-41cl krb5kdc[11483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:07:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:07:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:07:51.845628 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:35617
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:35617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:52.163986 11496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:52.164331 11496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:52.164559 11496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:52.172947 11496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:52.173094 11496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:52.173194 11496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:52.173303 11496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:52.173372 11496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:07:52.183337 11496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:35617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:35617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:52.184841 11496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:52.186128 11496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:52.197798 11501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:52.197929 11504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:52.197851 11502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:52.198567 11496 server_base.cc:1034] running on GCE node
I20260929 01:07:52.199314 11496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:52.200726 11496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:52.201946 11496 hybrid_clock.cc:648] HybridClock initialized: now 1790644072201901 us; error 42 us; skew 500 ppm
I20260929 01:07:52.202301 11496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:52 dist-test-slave-41cl krb5kdc[11483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644072, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:52.205469 11496 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:07:52.206872 11496 webserver.cc:460] Webserver started at http://127.5.198.254:32789/ using document root <none> and password file <none>
I20260929 01:07:52.207512 11496 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:52.207608 11496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:52.207858 11496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:52.210711 11496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "cb764116277647ef97c94779079b1492"
format_stamp: "Formatted at 2026-09-29 01:07:52 on dist-test-slave-41cl"
server_key: "42d3cfddc41ea04f68b438ce9bdd9449"
server_key_iv: "dfe737ea286ce4994f7aeb40605b10b0"
server_key_version: "encryptionkey@0"
I20260929 01:07:52.211432 11496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "cb764116277647ef97c94779079b1492"
format_stamp: "Formatted at 2026-09-29 01:07:52 on dist-test-slave-41cl"
server_key: "42d3cfddc41ea04f68b438ce9bdd9449"
server_key_iv: "dfe737ea286ce4994f7aeb40605b10b0"
server_key_version: "encryptionkey@0"
I20260929 01:07:52.216451 11496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 01:07:52.219718 11511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:52.221233 11496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:07:52.221436 11496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "cb764116277647ef97c94779079b1492"
format_stamp: "Formatted at 2026-09-29 01:07:52 on dist-test-slave-41cl"
server_key: "42d3cfddc41ea04f68b438ce9bdd9449"
server_key_iv: "dfe737ea286ce4994f7aeb40605b10b0"
server_key_version: "encryptionkey@0"
I20260929 01:07:52.221612 11496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:52.244390 11496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:52.253679 11496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:52.253970 11496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:52.272179 11496 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:35617
I20260929 01:07:52.272222 11562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:35617 every 8 connection(s)
I20260929 01:07:52.273464 11496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:07:52.276134 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 11496
I20260929 01:07:52.276348 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:07:52.277158 5915 external_mini_cluster.cc:1428] Setting key 68f9e5f7ee348a65429e12e4b1f7be63
I20260929 01:07:52.277679 11563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:52.291749 11563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492: Bootstrap starting.
Sep 29 01:07:52 dist-test-slave-41cl krb5kdc[11483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:52.294904 11563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:52.295939 11563 log.cc:824] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:52.298593 11563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492: No bootstrap required, opened a new log
I20260929 01:07:52.300709 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:52.279224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.5.198.254:35617)
0929 01:07:52.279992 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:52.280002 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:52.280040 (+ 38us) server_negotiation.cc:407] Connection header received
0929 01:07:52.282265 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:52.282287 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:52.282671 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:52.283004 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:52.284856 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:52.286183 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:52.287925 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:52.288680 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:52.288806 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:52.293391 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:52.293398 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:52.293411 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:52.293447 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:52.296619 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:52.297254 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:52.297263 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:52.297276 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:52.297423 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:52.297778 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:52.297784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:52.297787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:52.298242 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:52.298509 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:52.298962 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 01:07:52.299430 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":355,"threads_started":1}
I20260929 01:07:52.306383 11563 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb764116277647ef97c94779079b1492" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 35617 } }
I20260929 01:07:52.306679 11563 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:52.306747 11563 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb764116277647ef97c94779079b1492, State: Initialized, Role: FOLLOWER
I20260929 01:07:52.307272 11563 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb764116277647ef97c94779079b1492" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 35617 } }
I20260929 01:07:52.307461 11563 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:07:52.307554 11563 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:07:52.307703 11563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:52.309758 11563 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb764116277647ef97c94779079b1492" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 35617 } }
I20260929 01:07:52.310227 11563 leader_election.cc:302] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb764116277647ef97c94779079b1492; no voters:
I20260929 01:07:52.310796 11563 leader_election.cc:288] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:07:52.310933 11568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:52.311605 11568 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [term 1 LEADER]: Becoming Leader. State: Replica: cb764116277647ef97c94779079b1492, State: Running, Role: LEADER
I20260929 01:07:52.312122 11568 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb764116277647ef97c94779079b1492" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 35617 } }
I20260929 01:07:52.312961 11563 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:52.316190 11570 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb764116277647ef97c94779079b1492. Latest consensus state: current_term: 1 leader_uuid: "cb764116277647ef97c94779079b1492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb764116277647ef97c94779079b1492" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 35617 } } }
I20260929 01:07:52.316665 11570 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:52.317294 11569 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb764116277647ef97c94779079b1492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb764116277647ef97c94779079b1492" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 35617 } } }
I20260929 01:07:52.317689 11569 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:52.317679 11576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:07:52.323604 11576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:07:52.331753 11576 catalog_manager.cc:1347] Generated new cluster ID: c910aefa904d4ffe94fef55dd0eca5e3
I20260929 01:07:52.331871 11576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:07:52.341714 11576 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:07:52.343588 11576 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:07:52.350732 11576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb764116277647ef97c94779079b1492: Generated new TSK 0
I20260929 01:07:52.351694 11576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:07:52.404192 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:35617
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:52.726456 11589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:52.726788 11589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:52.726950 11589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:52.734036 11589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:52.734177 11589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:52.734342 11589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:07:52.743633 11589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:35617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:52.744864 11589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:52.746102 11589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:52.758504 11594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:52.758786 11595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:52.758786 11597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:52.759850 11589 server_base.cc:1034] running on GCE node
I20260929 01:07:52.760390 11589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:52.761289 11589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:52.762478 11589 hybrid_clock.cc:648] HybridClock initialized: now 1790644072762424 us; error 49 us; skew 500 ppm
I20260929 01:07:52.762760 11589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:52 dist-test-slave-41cl krb5kdc[11483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644072, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:52.765915 11589 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:07:52.767247 11589 webserver.cc:460] Webserver started at http://127.5.198.193:34399/ using document root <none> and password file <none>
I20260929 01:07:52.767901 11589 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:52.768044 11589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:52.768352 11589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:52.771147 11589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "e2489155caf048d586a6fbae58f58e9e"
format_stamp: "Formatted at 2026-09-29 01:07:52 on dist-test-slave-41cl"
server_key: "807b2d2b8f73dd351c1c9a714a39f749"
server_key_iv: "1826d8d4ab3e50db69dc6c5f8809562c"
server_key_version: "encryptionkey@0"
I20260929 01:07:52.771910 11589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "e2489155caf048d586a6fbae58f58e9e"
format_stamp: "Formatted at 2026-09-29 01:07:52 on dist-test-slave-41cl"
server_key: "807b2d2b8f73dd351c1c9a714a39f749"
server_key_iv: "1826d8d4ab3e50db69dc6c5f8809562c"
server_key_version: "encryptionkey@0"
I20260929 01:07:52.776985 11589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 01:07:52.780206 11604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:52.781663 11589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:07:52.781901 11589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "e2489155caf048d586a6fbae58f58e9e"
format_stamp: "Formatted at 2026-09-29 01:07:52 on dist-test-slave-41cl"
server_key: "807b2d2b8f73dd351c1c9a714a39f749"
server_key_iv: "1826d8d4ab3e50db69dc6c5f8809562c"
server_key_version: "encryptionkey@0"
I20260929 01:07:52.782132 11589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:52.812846 11589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:52.817514 11589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:52.817826 11589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:52.818836 11589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:52.820233 11589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:52.820348 11589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:52.820526 11589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:52.820621 11589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 01:07:52.853351 11589 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:44027
I20260929 01:07:52.853459 11716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:44027 every 8 connection(s)
I20260929 01:07:52.854655 11589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:07:52.856174 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 11589
I20260929 01:07:52.856400 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:07:52.857065 5915 external_mini_cluster.cc:1428] Setting key aa510701a559f71f3636b05b6013dd63
Sep 29 01:07:52 dist-test-slave-41cl krb5kdc[11483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644072, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:52.877158 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:52.857631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:41055 (local address 127.5.198.254:35617)
0929 01:07:52.858132 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:52.858139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:52.859246 (+ 1107us) server_negotiation.cc:407] Connection header received
0929 01:07:52.862017 (+ 2771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:52.862020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:52.862093 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:52.862184 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:52.864331 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:52.865177 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:52.867024 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:52.867453 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:52.867558 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:52.871547 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:52.871552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:52.871555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:52.871584 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:52.874268 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:52.874940 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:52.874947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:52.874950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:52.875063 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:52.875474 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:52.875481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:52.875484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:52.875724 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:52.875903 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:52.876586 (+ 683us) server_negotiation.cc:299] Negotiation successful
0929 01:07:52.876793 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":268,"threads_started":1}
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
I20260929 01:07:52.878103 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:52.858242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:35617 (local address 127.5.198.193:41055)
0929 01:07:52.859071 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:52.859126 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:52.861731 (+ 2605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:52.862437 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:52.862452 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:52.863197 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:52.863992 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:52.864020 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:52.865356 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:52.865364 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:52.866822 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:52.866835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:52.867671 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:52.867680 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:52.868005 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:52.868786 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:52.868819 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:52.871345 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:52.874397 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:52.874406 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:52.874427 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:52.874822 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:52.875171 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:52.875177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:52.875180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:52.875362 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:52.875868 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:52.875878 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:52.876191 (+ 313us) client_negotiation.cc:769] Sending connection context
0929 01:07:52.876446 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 01:07:52.876963 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":306,"threads_started":1}
I20260929 01:07:52.879606 11717 heartbeater.cc:347] Connected to a master server at 127.5.198.254:35617
I20260929 01:07:52.879917 11717 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
I20260929 01:07:52.880975 11717 heartbeater.cc:511] Master 127.5.198.254:35617 requested a full tablet report, sending...
I20260929 01:07:52.883169 11528 ts_manager.cc:194] Registered new tserver with Master: e2489155caf048d586a6fbae58f58e9e (127.5.198.193:44027)
I20260929 01:07:52.884784 11528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:41055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:07:52.904403 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:35617
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:53.228456 11723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:53.228731 11723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:53.228822 11723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:53.235867 11723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:53.235980 11723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:53.236090 11723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:07:53.245589 11723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:35617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:53.246763 11723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:53.248027 11723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:53.260958 11728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:53.261073 11731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:53.261096 11729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:53.261706 11723 server_base.cc:1034] running on GCE node
I20260929 01:07:53.262296 11723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:53.263149 11723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:53.264341 11723 hybrid_clock.cc:648] HybridClock initialized: now 1790644073264291 us; error 43 us; skew 500 ppm
I20260929 01:07:53.264681 11723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:53 dist-test-slave-41cl krb5kdc[11483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644073, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:53.267724 11723 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:07:53.269085 11723 webserver.cc:460] Webserver started at http://127.5.198.194:44917/ using document root <none> and password file <none>
I20260929 01:07:53.269695 11723 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:53.269793 11723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:53.270071 11723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:53.273471 11723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "c7b64f7978a343308a517cfc185ffe19"
format_stamp: "Formatted at 2026-09-29 01:07:53 on dist-test-slave-41cl"
server_key: "ee8cb58f0b6d7894ec75d08d4e351067"
server_key_iv: "a544f510466744388df624db70d8ee09"
server_key_version: "encryptionkey@0"
I20260929 01:07:53.274432 11723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "c7b64f7978a343308a517cfc185ffe19"
format_stamp: "Formatted at 2026-09-29 01:07:53 on dist-test-slave-41cl"
server_key: "ee8cb58f0b6d7894ec75d08d4e351067"
server_key_iv: "a544f510466744388df624db70d8ee09"
server_key_version: "encryptionkey@0"
I20260929 01:07:53.280664 11723 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:07:53.284855 11738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:53.286350 11723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:07:53.286528 11723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "c7b64f7978a343308a517cfc185ffe19"
format_stamp: "Formatted at 2026-09-29 01:07:53 on dist-test-slave-41cl"
server_key: "ee8cb58f0b6d7894ec75d08d4e351067"
server_key_iv: "a544f510466744388df624db70d8ee09"
server_key_version: "encryptionkey@0"
I20260929 01:07:53.286733 11723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:53.303138 11723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:53.307116 11723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:53.307395 11723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:53.308298 11723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:53.309604 11723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:53.309705 11723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:53.309859 11723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:53.309931 11723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:53.344405 11723 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:45731
I20260929 01:07:53.344441 11850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:45731 every 8 connection(s)
I20260929 01:07:53.345660 11723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:07:53.355530 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 11723
I20260929 01:07:53.355695 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:07:53.356251 5915 external_mini_cluster.cc:1428] Setting key c4a69fa5214752bec65ffaa7641f3a4d
Sep 29 01:07:53 dist-test-slave-41cl krb5kdc[11483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644073, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:53.370885 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:53.348170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:42825 (local address 127.5.198.254:35617)
0929 01:07:53.348338 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:53.348343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:53.349657 (+ 1314us) server_negotiation.cc:407] Connection header received
0929 01:07:53.351432 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:53.351437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:53.351526 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:53.351636 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:53.353164 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:53.354027 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:53.356295 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:53.356956 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:53.357036 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:53.362302 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:53.362310 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:53.362318 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:53.362363 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:53.366462 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:53.367772 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:53.367781 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:53.367786 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:53.368021 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:53.368611 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:53.368617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:53.368621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:53.369185 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:53.369364 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:53.370094 (+ 730us) server_negotiation.cc:299] Negotiation successful
0929 01:07:53.370484 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:07:53.372195 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:53.348742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:35617 (local address 127.5.198.194:42825)
0929 01:07:53.349506 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:53.349543 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:53.351160 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:53.351863 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:53.351872 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:53.352351 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:53.352995 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:53.353014 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:53.354185 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:53.354189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:53.356123 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:53.356137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:53.357355 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:53.357364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:53.357608 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:53.358372 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:53.358400 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:53.362050 (+ 3650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:53.366631 (+ 4581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:53.366643 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:53.366662 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:53.367574 (+ 912us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:53.368206 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:53.368213 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:53.368219 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:53.368442 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:53.369334 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:53.369343 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:53.369711 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 01:07:53.370005 (+ 294us) client_negotiation.cc:240] Negotiation successful
0929 01:07:53.370774 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":310,"threads_started":1}
I20260929 01:07:53.373842 11851 heartbeater.cc:347] Connected to a master server at 127.5.198.254:35617
I20260929 01:07:53.374173 11851 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:53.375078 11851 heartbeater.cc:511] Master 127.5.198.254:35617 requested a full tablet report, sending...
I20260929 01:07:53.376852 11528 ts_manager.cc:194] Registered new tserver with Master: c7b64f7978a343308a517cfc185ffe19 (127.5.198.194:45731)
I20260929 01:07:53.378123 11528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:42825
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:07:53.416137 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:35617
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:53.732893 11856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:53.733161 11856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:53.733307 11856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:53.740586 11856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:53.740689 11856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:53.740881 11856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:07:53.750397 11856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:35617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:53.751628 11856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:53.752948 11856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:53.765798 11861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:53.766062 11862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:53.766191 11864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:53.772365 11856 server_base.cc:1034] running on GCE node
I20260929 01:07:53.773171 11856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:53.774178 11856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:53.775425 11856 hybrid_clock.cc:648] HybridClock initialized: now 1790644073775389 us; error 52 us; skew 500 ppm
I20260929 01:07:53.775764 11856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:53 dist-test-slave-41cl krb5kdc[11483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644073, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:53.779116 11856 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:07:53.780664 11856 webserver.cc:460] Webserver started at http://127.5.198.195:39323/ using document root <none> and password file <none>
I20260929 01:07:53.781363 11856 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:53.781502 11856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:53.781831 11856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:53.784687 11856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "9414b00af2904515905c3da8d6383f0c"
format_stamp: "Formatted at 2026-09-29 01:07:53 on dist-test-slave-41cl"
server_key: "a335187cb87dfe88357f51b3d1f9bc1e"
server_key_iv: "5ef34872fff100476e32ab9234d65408"
server_key_version: "encryptionkey@0"
I20260929 01:07:53.785445 11856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "9414b00af2904515905c3da8d6383f0c"
format_stamp: "Formatted at 2026-09-29 01:07:53 on dist-test-slave-41cl"
server_key: "a335187cb87dfe88357f51b3d1f9bc1e"
server_key_iv: "5ef34872fff100476e32ab9234d65408"
server_key_version: "encryptionkey@0"
I20260929 01:07:53.790629 11856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:07:53.794461 11871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:53.795855 11856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:07:53.796059 11856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "9414b00af2904515905c3da8d6383f0c"
format_stamp: "Formatted at 2026-09-29 01:07:53 on dist-test-slave-41cl"
server_key: "a335187cb87dfe88357f51b3d1f9bc1e"
server_key_iv: "5ef34872fff100476e32ab9234d65408"
server_key_version: "encryptionkey@0"
I20260929 01:07:53.796264 11856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:53.826449 11856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:53.830394 11856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:53.830682 11856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:53.831648 11856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:53.832991 11856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:53.833092 11856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:53.833237 11856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:53.833316 11856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:53.871160 11856 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:36029
I20260929 01:07:53.871207 11983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:36029 every 8 connection(s)
I20260929 01:07:53.872519 11856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:07:53.877169 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 11856
I20260929 01:07:53.877346 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:07:53.877772 5915 external_mini_cluster.cc:1428] Setting key 891f32569257d4a21f557b99fbd39634
Sep 29 01:07:53 dist-test-slave-41cl krb5kdc[11483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644073, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:07:53.888137 11717 heartbeater.cc:503] Master 127.5.198.254:35617 was elected leader, sending a full tablet report...
I20260929 01:07:53.893455 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:53.875179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:41559 (local address 127.5.198.254:35617)
0929 01:07:53.875614 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:53.875619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:53.876629 (+ 1010us) server_negotiation.cc:407] Connection header received
0929 01:07:53.878538 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:53.878544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:53.878633 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:53.878749 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:53.880460 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:53.881315 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:53.882908 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:53.883302 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:53.883424 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:53.887012 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:53.887017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:53.887021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:53.887055 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:53.890186 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:53.891005 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:53.891012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:53.891016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:53.891165 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:53.891637 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:53.891642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:53.891645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:53.892001 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:53.892141 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:53.893034 (+ 893us) server_negotiation.cc:299] Negotiation successful
0929 01:07:53.893199 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":218,"threads_started":1}
I20260929 01:07:53.894498 11987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:53.875745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:35617 (local address 127.5.198.195:41559)
0929 01:07:53.876406 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:53.876442 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:53.878269 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:53.879083 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:53.879092 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:53.879660 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:53.880282 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:53.880308 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:53.881448 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:53.881452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:53.882731 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:53.882743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:53.883535 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:53.883541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:53.883745 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:53.884368 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:53.884390 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:53.886786 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:53.890358 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:53.890366 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:53.890386 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:53.890855 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:53.891292 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:53.891297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:53.891301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:53.891552 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:53.892139 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:53.892147 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:53.892616 (+ 469us) client_negotiation.cc:769] Sending connection context
0929 01:07:53.892921 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 01:07:53.893380 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":277,"threads_started":1}
I20260929 01:07:53.895816 11984 heartbeater.cc:347] Connected to a master server at 127.5.198.254:35617
I20260929 01:07:53.896133 11984 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:53.896939 11984 heartbeater.cc:511] Master 127.5.198.254:35617 requested a full tablet report, sending...
I20260929 01:07:53.898267 11528 ts_manager.cc:194] Registered new tserver with Master: 9414b00af2904515905c3da8d6383f0c (127.5.198.195:36029)
I20260929 01:07:53.899061 11528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:41559
I20260929 01:07:53.904987 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/test-admin.keytab.
Sep 29 01:07:53 dist-test-slave-41cl krb5kdc[11483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:53.933517 5915 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 01:07:53.938498 11992 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:07:53.936102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.5.198.193:44027)
0929 01:07:53.936777 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:53.936784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:53.936810 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:07:53.936870 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:53.936876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:53.937125 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:53.937250 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:53.938105 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39956: BlockingRecv error: recv got EOF from 127.0.0.1:39956 (error 108)
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":342,"threads_started":1}
W20260929 01:07:53.938676 5915 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/krb5cc)
Sep 29 01:07:53 dist-test-slave-41cl krb5kdc[11483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:53.940163 5915 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 01:07:53 dist-test-slave-41cl krb5kdc[11483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:53.941485 5915 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 01:07:53 dist-test-slave-41cl krb5kdc[11483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:07:53.956720 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:53.943078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39970 (local address 127.5.198.193:44027)
0929 01:07:53.943267 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:53.943272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:53.943399 (+ 127us) server_negotiation.cc:407] Connection header received
0929 01:07:53.943530 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:53.943533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:53.943588 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:53.943666 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:53.944617 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:53.945555 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:53.946790 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:53.947331 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:53.947482 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:53.950948 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:53.950960 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:53.950973 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:53.951007 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:53.953768 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:53.954485 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:53.954494 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:53.954497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:53.954629 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:53.955027 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:53.955034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:53.955037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:53.955402 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:53.955706 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:53.956056 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 01:07:53.956333 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 01:07:53.962508 11992 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:07:53.960871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39980 (local address 127.5.198.193:44027)
0929 01:07:53.961037 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:53.961041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:53.961367 (+ 326us) server_negotiation.cc:407] Connection header received
0929 01:07:53.961493 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:53.961496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:53.961549 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:53.961625 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:53.962218 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39980: BlockingRecv error: recv got EOF from 127.0.0.1:39980 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 01:07:53.962767 5915 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/krb5cc' of type FILE
Sep 29 01:07:53 dist-test-slave-41cl krb5kdc[11483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:53.964218 5915 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 01:07:53 dist-test-slave-41cl krb5kdc[11483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:53.965550 5915 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 01:07:53 dist-test-slave-41cl krb5kdc[11483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:07:53.979099 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:53.966754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.5.198.193:44027)
0929 01:07:53.966983 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:53.966989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:53.967208 (+ 219us) server_negotiation.cc:407] Connection header received
0929 01:07:53.967386 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:53.967391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:53.967457 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:53.967553 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:53.968397 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:53.969138 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:53.970312 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:53.970738 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:53.970867 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:53.974238 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:53.974244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:53.974249 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:53.974301 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:53.976699 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:53.977313 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:53.977319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:53.977323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:53.977444 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:53.977810 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:53.977817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:53.977820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:53.978085 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:53.978349 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:53.978622 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 01:07:53.978816 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 01:07:53.983139 11992 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:07:53.981758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.5.198.193:44027)
0929 01:07:53.981944 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:53.981949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:53.982019 (+ 70us) server_negotiation.cc:407] Connection header received
0929 01:07:53.982132 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:53.982135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:53.982199 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:53.982272 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:53.982904 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39988: BlockingRecv error: recv got EOF from 127.0.0.1:39988 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 01:07:53.983260 5915 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790644023292677-5915-0/krb5kdc/krb5cc)
Sep 29 01:07:53 dist-test-slave-41cl krb5kdc[11483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:53.984805 5915 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 01:07:53 dist-test-slave-41cl krb5kdc[11483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:53.986248 5915 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 01:07:53 dist-test-slave-41cl krb5kdc[11483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:07:53.999881 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:53.987523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39992 (local address 127.5.198.193:44027)
0929 01:07:53.987728 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:53.987735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:53.987984 (+ 249us) server_negotiation.cc:407] Connection header received
0929 01:07:53.988154 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:53.988159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:53.988221 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:53.988319 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:53.989344 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:53.990041 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:53.991205 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:53.991617 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:53.991749 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:53.995109 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:53.995113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:53.995116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:53.995145 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:53.997644 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:53.998203 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:53.998207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:53.998209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:53.998339 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:53.998628 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:53.998631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:53.998633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:53.998859 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:53.999069 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:53.999285 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 01:07:53.999465 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:07:54.001511 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 11589
I20260929 01:07:54.012569 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:54.002213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.5.198.193:44027)
0929 01:07:54.002353 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:54.002356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:54.002730 (+ 374us) server_negotiation.cc:407] Connection header received
0929 01:07:54.002846 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:54.002849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:54.002899 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:54.002973 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:54.003792 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:54.004420 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:54.005624 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:54.006022 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:54.006112 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:54.007961 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:54.007965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:54.007968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:54.007995 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:54.010434 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:54.011010 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:54.011013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:54.011016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:54.011125 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:54.011450 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:54.011452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:54.011454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:54.011679 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:54.011857 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:54.012102 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 01:07:54.012245 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 01:07:54.013150 11712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:54.094791 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 11589
I20260929 01:07:54.111135 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 11723
Sep 29 01:07:54 dist-test-slave-41cl krb5kdc[11483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:07:54.125900 12005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:54.112010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.5.198.194:45731)
0929 01:07:54.112572 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:54.112579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:54.112604 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:07:54.112666 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:54.112673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:54.112872 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:54.112999 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:54.114124 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:54.115088 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:54.116225 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:54.116917 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:54.117016 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:54.120663 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:54.120674 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:54.120687 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:54.120718 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:54.123469 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:54.124091 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:54.124098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:54.124100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:54.124213 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:54.124587 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:54.124594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:54.124599 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:54.124944 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:54.125248 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:54.125379 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 01:07:54.125612 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":237,"threads_started":1}
I20260929 01:07:54.127193 11846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:54.208415 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 11723
I20260929 01:07:54.224303 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 11856
Sep 29 01:07:54 dist-test-slave-41cl krb5kdc[11483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:07:54.241015 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:54.225446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54300 (local address 127.5.198.195:36029)
0929 01:07:54.226098 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:54.226105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:54.226134 (+ 29us) server_negotiation.cc:407] Connection header received
0929 01:07:54.226192 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:54.226198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:54.226540 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:54.226766 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:54.227872 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:54.229001 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:54.230345 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:54.230905 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:54.231049 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:54.235134 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:54.235145 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:54.235160 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:54.235194 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:54.238167 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:54.238923 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:54.238930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:54.238932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:54.239059 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:54.239444 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:54.239448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:54.239450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:54.239835 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:54.240093 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:54.240366 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 01:07:54.240662 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":350,"threads_started":1}
I20260929 01:07:54.242403 11979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:54.324265 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 11856
I20260929 01:07:54.341094 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 11496
I20260929 01:07:54.342280 11558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:54.444399 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 11496
2026-09-29T01:07:54Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3900 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:07:54 dist-test-slave-41cl krb5kdc[12012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:07:54 dist-test-slave-41cl krb5kdc[12012](info): set up 2 sockets
Sep 29 01:07:54 dist-test-slave-41cl krb5kdc[12012](info): commencing operation
krb5kdc: starting...
W20260929 01:07:55.565213 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:07:55 dist-test-slave-41cl krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:07:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:07:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.254.keytab.
Entry for principal oryx/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.254.keytab.
I20260929 01:07:55.772260 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:39173
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.254.keytab
--principal=oryx/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:39173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:56.088455 12025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:56.088835 12025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:56.088992 12025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:56.095992 12025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:56.096153 12025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:56.096254 12025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:56.096355 12025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:56.096431 12025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:07:56.106818 12025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:39173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.254.keytab
--principal=oryx/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:39173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:56.108170 12025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:56.109468 12025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:56.121104 12033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:56.121176 12030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:56.121152 12031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:56.121780 12025 server_base.cc:1034] running on GCE node
I20260929 01:07:56.122560 12025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:56.123853 12025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:56.125039 12025 hybrid_clock.cc:648] HybridClock initialized: now 1790644076124988 us; error 50 us; skew 500 ppm
I20260929 01:07:56.125363 12025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:56 dist-test-slave-41cl krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644076, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:56.128276 12025 init.cc:377] Logged in from keytab as oryx/127.5.198.254@KRBTEST.COM (short username oryx)
I20260929 01:07:56.129760 12025 webserver.cc:460] Webserver started at http://127.5.198.254:41525/ using document root <none> and password file <none>
I20260929 01:07:56.130363 12025 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:56.130460 12025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:56.130702 12025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:56.133510 12025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "03087ec95dd74fec8f8ccd4af29de874"
format_stamp: "Formatted at 2026-09-29 01:07:56 on dist-test-slave-41cl"
server_key: "c561fd606e967cd51e9d241de6461285"
server_key_iv: "b67d2c713d42d235b5e9eaffde06c6b7"
server_key_version: "encryptionkey@0"
I20260929 01:07:56.134315 12025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "03087ec95dd74fec8f8ccd4af29de874"
format_stamp: "Formatted at 2026-09-29 01:07:56 on dist-test-slave-41cl"
server_key: "c561fd606e967cd51e9d241de6461285"
server_key_iv: "b67d2c713d42d235b5e9eaffde06c6b7"
server_key_version: "encryptionkey@0"
I20260929 01:07:56.139169 12025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 01:07:56.142474 12040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:56.143726 12025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:07:56.143898 12025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "03087ec95dd74fec8f8ccd4af29de874"
format_stamp: "Formatted at 2026-09-29 01:07:56 on dist-test-slave-41cl"
server_key: "c561fd606e967cd51e9d241de6461285"
server_key_iv: "b67d2c713d42d235b5e9eaffde06c6b7"
server_key_version: "encryptionkey@0"
I20260929 01:07:56.144114 12025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:56.173745 12025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:56.177412 12025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:56.177687 12025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:56.195240 12025 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:39173
I20260929 01:07:56.195271 12091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:39173 every 8 connection(s)
I20260929 01:07:56.196535 12025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:07:56.199935 12092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:56.200995 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 12025
I20260929 01:07:56.201191 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:07:56.201809 5915 external_mini_cluster.cc:1428] Setting key ef4bd74a44bc56ff34b70e37cc6c38af
I20260929 01:07:56.213611 12092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874: Bootstrap starting.
Sep 29 01:07:56 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM
I20260929 01:07:56.216739 12092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:56.218075 12092 log.cc:824] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:56.221280 12092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874: No bootstrap required, opened a new log
I20260929 01:07:56.225019 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:56.203690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.5.198.254:39173)
0929 01:07:56.204439 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:56.204453 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:56.204527 (+ 74us) server_negotiation.cc:407] Connection header received
0929 01:07:56.206186 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:56.206212 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:56.206689 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:56.207118 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:56.208391 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:56.210351 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:56.211652 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:56.212676 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:56.212847 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:56.216935 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:56.216947 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:56.216964 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:56.217014 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:56.220748 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:56.221396 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:56.221404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:56.221411 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:56.221609 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:56.221954 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:56.221961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:56.221966 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:56.222624 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:56.222947 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:56.223209 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 01:07:56.223647 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":318,"threads_started":1}
I20260929 01:07:56.230877 12092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } }
I20260929 01:07:56.231207 12092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:56.231276 12092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03087ec95dd74fec8f8ccd4af29de874, State: Initialized, Role: FOLLOWER
I20260929 01:07:56.231745 12092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } }
I20260929 01:07:56.231933 12092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:07:56.232025 12092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:07:56.232146 12092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:56.234428 12092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } }
I20260929 01:07:56.234841 12092 leader_election.cc:302] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03087ec95dd74fec8f8ccd4af29de874; no voters:
I20260929 01:07:56.235359 12092 leader_election.cc:288] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:07:56.235497 12097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:07:56.236142 12097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [term 1 LEADER]: Becoming Leader. State: Replica: 03087ec95dd74fec8f8ccd4af29de874, State: Running, Role: LEADER
I20260929 01:07:56.236675 12097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } }
I20260929 01:07:56.237457 12092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:56.239336 12098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03087ec95dd74fec8f8ccd4af29de874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } }
I20260929 01:07:56.239629 12098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:56.241641 12099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03087ec95dd74fec8f8ccd4af29de874. Latest consensus state: current_term: 1 leader_uuid: "03087ec95dd74fec8f8ccd4af29de874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } }
I20260929 01:07:56.242027 12099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:56.243335 12103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:07:56.248807 12103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:07:56.257026 12103 catalog_manager.cc:1347] Generated new cluster ID: 55239c9c72aa4bf08cacc386bfc02c32
I20260929 01:07:56.257129 12103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:07:56.268623 12103 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:07:56.270417 12103 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:07:56.284512 12103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874: Generated new TSK 0
I20260929 01:07:56.285516 12103 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.193.keytab.
Entry for principal oryx/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.193.keytab.
I20260929 01:07:56.340715 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:39173
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.193.keytab
--principal=oryx/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:56.659379 12118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:56.659643 12118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:56.659832 12118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:56.666918 12118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:56.667017 12118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:56.667210 12118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:07:56.677150 12118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.193.keytab
--principal=oryx/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:39173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:56.678452 12118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:56.679838 12118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:56.693308 12123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:56.693276 12124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:56.693415 12126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:56.694257 12118 server_base.cc:1034] running on GCE node
I20260929 01:07:56.694952 12118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:56.695878 12118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:56.697135 12118 hybrid_clock.cc:648] HybridClock initialized: now 1790644076697072 us; error 58 us; skew 500 ppm
I20260929 01:07:56.697443 12118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:56 dist-test-slave-41cl krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644076, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:56.700735 12118 init.cc:377] Logged in from keytab as oryx/127.5.198.193@KRBTEST.COM (short username oryx)
I20260929 01:07:56.702145 12118 webserver.cc:460] Webserver started at http://127.5.198.193:37325/ using document root <none> and password file <none>
I20260929 01:07:56.702816 12118 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:56.702979 12118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:56.703272 12118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:56.706143 12118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "4cd3434ac2e041c299afa3fcfd3031ac"
format_stamp: "Formatted at 2026-09-29 01:07:56 on dist-test-slave-41cl"
server_key: "c6bec09be8de2aa0d34c954885d67c5b"
server_key_iv: "9c3ff93b960cfc161c5888b968ad9b72"
server_key_version: "encryptionkey@0"
I20260929 01:07:56.706892 12118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "4cd3434ac2e041c299afa3fcfd3031ac"
format_stamp: "Formatted at 2026-09-29 01:07:56 on dist-test-slave-41cl"
server_key: "c6bec09be8de2aa0d34c954885d67c5b"
server_key_iv: "9c3ff93b960cfc161c5888b968ad9b72"
server_key_version: "encryptionkey@0"
I20260929 01:07:56.712129 12118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:07:56.715750 12133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:56.717253 12118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:07:56.717434 12118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "4cd3434ac2e041c299afa3fcfd3031ac"
format_stamp: "Formatted at 2026-09-29 01:07:56 on dist-test-slave-41cl"
server_key: "c6bec09be8de2aa0d34c954885d67c5b"
server_key_iv: "9c3ff93b960cfc161c5888b968ad9b72"
server_key_version: "encryptionkey@0"
I20260929 01:07:56.717636 12118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:56.759040 12118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:56.763026 12118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:56.763322 12118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:56.764247 12118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:56.765609 12118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:56.765712 12118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:56.765861 12118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:56.765939 12118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:56.798400 12118 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:35399
I20260929 01:07:56.798486 12245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:35399 every 8 connection(s)
I20260929 01:07:56.799677 12118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:07:56.801879 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 12118
I20260929 01:07:56.802081 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:07:56.802838 5915 external_mini_cluster.cc:1428] Setting key ec94eab1c2f4008af966bf62affc5671
Sep 29 01:07:56 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644076, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM
I20260929 01:07:56.823313 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:56.802455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:43497 (local address 127.5.198.254:39173)
0929 01:07:56.802930 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:56.802937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:56.803946 (+ 1009us) server_negotiation.cc:407] Connection header received
0929 01:07:56.806419 (+ 2473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:56.806423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:56.806483 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:56.806573 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:56.808837 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:56.809560 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:56.812024 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:56.812717 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:56.812859 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:56.816349 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:56.816358 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:56.816367 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:56.816410 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:56.819883 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:56.820596 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:56.820603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:56.820610 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:56.820755 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:56.821252 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:56.821259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:56.821265 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:56.821644 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:56.821802 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:56.822583 (+ 781us) server_negotiation.cc:299] Negotiation successful
0929 01:07:56.822850 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":209,"threads_started":1}
I20260929 01:07:56.823858 12249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:56.803038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.193:43497)
0929 01:07:56.803782 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:56.803834 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:56.806124 (+ 2290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:56.806858 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:56.806873 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:56.807764 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:56.808641 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:56.808669 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:56.809713 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:56.809721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:56.811866 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:56.811879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:56.812921 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:56.812926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:56.813116 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:56.813789 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:56.813810 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:56.816129 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:56.820031 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:56.820038 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:56.820049 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:56.820447 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:56.820875 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:56.820879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:56.820881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:56.821116 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:56.821826 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:56.821831 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:56.822199 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 01:07:56.822412 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 01:07:56.822775 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":298,"threads_started":1}
I20260929 01:07:56.825387 12246 heartbeater.cc:347] Connected to a master server at 127.5.198.254:39173
WARNING: no policy specified for oryx/127.5.198.194@KRBTEST.COM; defaulting to no policy
I20260929 01:07:56.825733 12246 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:56.826494 12246 heartbeater.cc:511] Master 127.5.198.254:39173 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.198.194@KRBTEST.COM" created.
I20260929 01:07:56.828500 12057 ts_manager.cc:194] Registered new tserver with Master: 4cd3434ac2e041c299afa3fcfd3031ac (127.5.198.193:35399)
I20260929 01:07:56.830626 12057 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:43497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.194.keytab.
Entry for principal oryx/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.194.keytab.
I20260929 01:07:56.852778 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:39173
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.194.keytab
--principal=oryx/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:57.169196 12252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:57.169476 12252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:57.169637 12252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:57.176894 12252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:57.176996 12252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:57.177194 12252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:07:57.186741 12252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.194.keytab
--principal=oryx/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:39173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:57.188027 12252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:57.189324 12252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:57.202246 12258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:57.202785 12260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:57.203194 12257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:57.203621 12252 server_base.cc:1034] running on GCE node
I20260929 01:07:57.204198 12252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:57.205165 12252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:57.206387 12252 hybrid_clock.cc:648] HybridClock initialized: now 1790644077206336 us; error 45 us; skew 500 ppm
I20260929 01:07:57.206691 12252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:57 dist-test-slave-41cl krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644077, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:57.209826 12252 init.cc:377] Logged in from keytab as oryx/127.5.198.194@KRBTEST.COM (short username oryx)
I20260929 01:07:57.211211 12252 webserver.cc:460] Webserver started at http://127.5.198.194:36431/ using document root <none> and password file <none>
I20260929 01:07:57.211855 12252 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:57.211951 12252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:57.212219 12252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:57.215009 12252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "618b7791d9e34150811fd7af3fb00c5a"
format_stamp: "Formatted at 2026-09-29 01:07:57 on dist-test-slave-41cl"
server_key: "5bf5cae55abf5b315c629fadefaab8f8"
server_key_iv: "0d3d829c55aa381d194880f1d3d0cc4f"
server_key_version: "encryptionkey@0"
I20260929 01:07:57.215718 12252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "618b7791d9e34150811fd7af3fb00c5a"
format_stamp: "Formatted at 2026-09-29 01:07:57 on dist-test-slave-41cl"
server_key: "5bf5cae55abf5b315c629fadefaab8f8"
server_key_iv: "0d3d829c55aa381d194880f1d3d0cc4f"
server_key_version: "encryptionkey@0"
I20260929 01:07:57.220717 12252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 01:07:57.223955 12267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:57.225193 12252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:07:57.225354 12252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "618b7791d9e34150811fd7af3fb00c5a"
format_stamp: "Formatted at 2026-09-29 01:07:57 on dist-test-slave-41cl"
server_key: "5bf5cae55abf5b315c629fadefaab8f8"
server_key_iv: "0d3d829c55aa381d194880f1d3d0cc4f"
server_key_version: "encryptionkey@0"
I20260929 01:07:57.225548 12252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:57.245123 12252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:57.249410 12252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:57.249704 12252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:57.250658 12252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:57.251891 12252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:57.252007 12252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:57.252148 12252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:57.252220 12252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:57.284339 12252 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:42701
I20260929 01:07:57.284389 12379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:42701 every 8 connection(s)
I20260929 01:07:57.285624 12252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:07:57.290966 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 12252
I20260929 01:07:57.291137 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:07:57.291563 5915 external_mini_cluster.cc:1428] Setting key 71dfe0cf7095711b7648b587c58092d2
Sep 29 01:07:57 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644077, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.5.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.198.195@KRBTEST.COM" created.
I20260929 01:07:57.314919 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:57.288243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:44725 (local address 127.5.198.254:39173)
0929 01:07:57.288414 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:57.288418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:57.289590 (+ 1172us) server_negotiation.cc:407] Connection header received
0929 01:07:57.291388 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:57.291394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:57.291485 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:57.291604 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:57.293235 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:57.294076 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:57.296585 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:57.296985 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:57.297073 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:57.308588 (+ 11515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:57.308597 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:57.308604 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:57.308644 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:57.311928 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:57.312747 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:57.312752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:57.312754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:57.312899 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:57.313331 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:57.313335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:57.313337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:57.313582 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:57.313688 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:57.314495 (+ 807us) server_negotiation.cc:299] Negotiation successful
0929 01:07:57.314665 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 01:07:57.316536 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:57.288727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.194:44725)
0929 01:07:57.289430 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:57.289471 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:57.291058 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:57.291836 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:57.291843 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:57.292407 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:57.293035 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:57.293052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:57.294228 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:57.294234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:57.296391 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:57.296407 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:57.297203 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:57.297212 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:57.297543 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:57.305886 (+ 8343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:57.305912 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:57.308297 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:57.312054 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:57.312061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:57.312074 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:57.312632 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:57.312996 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:57.313000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:57.313002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:57.313198 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:57.313683 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:57.313689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:57.314097 (+ 408us) client_negotiation.cc:769] Sending connection context
0929 01:07:57.314406 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 01:07:57.314995 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":262,"threads_started":1}
I20260929 01:07:57.318034 12380 heartbeater.cc:347] Connected to a master server at 127.5.198.254:39173
I20260929 01:07:57.318343 12380 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:57.319201 12380 heartbeater.cc:511] Master 127.5.198.254:39173 requested a full tablet report, sending...
I20260929 01:07:57.320779 12057 ts_manager.cc:194] Registered new tserver with Master: 618b7791d9e34150811fd7af3fb00c5a (127.5.198.194:42701)
I20260929 01:07:57.321708 12057 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:44725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.195.keytab.
Entry for principal oryx/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.195.keytab.
I20260929 01:07:57.340095 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:39173
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.195.keytab
--principal=oryx/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:57.652693 12385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:57.653018 12385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:57.653189 12385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:57.660142 12385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:57.660283 12385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:57.660486 12385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:07:57.669886 12385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.195.keytab
--principal=oryx/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:39173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:57.671180 12385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:57.672451 12385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:57.685168 12390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:57.685153 12391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:57.685258 12393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:57.686275 12385 server_base.cc:1034] running on GCE node
I20260929 01:07:57.686851 12385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:57.687664 12385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:57.688879 12385 hybrid_clock.cc:648] HybridClock initialized: now 1790644077688817 us; error 57 us; skew 500 ppm
I20260929 01:07:57.689188 12385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:57 dist-test-slave-41cl krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644077, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:57.692109 12385 init.cc:377] Logged in from keytab as oryx/127.5.198.195@KRBTEST.COM (short username oryx)
I20260929 01:07:57.693536 12385 webserver.cc:460] Webserver started at http://127.5.198.195:34235/ using document root <none> and password file <none>
I20260929 01:07:57.694196 12385 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:57.694340 12385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:57.694644 12385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:57.697477 12385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "59b3bc938cb34499a7aa7607b65ea443"
format_stamp: "Formatted at 2026-09-29 01:07:57 on dist-test-slave-41cl"
server_key: "f7794d243d99a652be18e79255ebde13"
server_key_iv: "d86918e034f793fb424b79c1e560bc11"
server_key_version: "encryptionkey@0"
I20260929 01:07:57.698203 12385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "59b3bc938cb34499a7aa7607b65ea443"
format_stamp: "Formatted at 2026-09-29 01:07:57 on dist-test-slave-41cl"
server_key: "f7794d243d99a652be18e79255ebde13"
server_key_iv: "d86918e034f793fb424b79c1e560bc11"
server_key_version: "encryptionkey@0"
I20260929 01:07:57.703259 12385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 01:07:57.706549 12400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:57.707819 12385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 01:07:57.708004 12385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "59b3bc938cb34499a7aa7607b65ea443"
format_stamp: "Formatted at 2026-09-29 01:07:57 on dist-test-slave-41cl"
server_key: "f7794d243d99a652be18e79255ebde13"
server_key_iv: "d86918e034f793fb424b79c1e560bc11"
server_key_version: "encryptionkey@0"
I20260929 01:07:57.708299 12385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:57.729280 12385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:57.733824 12385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:57.734109 12385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:57.735033 12385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:57.736327 12385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:57.736434 12385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:57.736629 12385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:57.736704 12385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:57.779811 12385 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:35401
I20260929 01:07:57.779853 12512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:35401 every 8 connection(s)
I20260929 01:07:57.781122 12385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:07:57.789278 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 12385
I20260929 01:07:57.789474 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:07:57.790112 5915 external_mini_cluster.cc:1428] Setting key dd53670e17b38c789432cdb87fc1f439
Sep 29 01:07:57 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644077, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM
I20260929 01:07:57.802433 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:57.784731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:42029 (local address 127.5.198.254:39173)
0929 01:07:57.784965 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:57.784970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:57.785600 (+ 630us) server_negotiation.cc:407] Connection header received
0929 01:07:57.787442 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:57.787447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:57.787548 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:57.787653 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:57.789144 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:57.790028 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:57.791877 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:57.792529 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:57.792655 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:57.796894 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:57.796899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:57.796909 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:57.796937 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:57.799553 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:57.800238 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:57.800248 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:57.800250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:57.800363 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:57.800779 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:57.800784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:57.800787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:57.801034 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:57.801181 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:57.801890 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 01:07:57.802093 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260929 01:07:57.803426 12515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:57.784676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.195:42029)
0929 01:07:57.785442 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:57.785479 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:57.787116 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:57.787914 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:57.787921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:57.788387 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:57.788984 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:57.789001 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:57.790228 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:57.790233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:57.791730 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:57.791742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:57.792761 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:57.792770 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:57.793123 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:57.793849 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:57.793885 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:57.796704 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:57.799688 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:57.799696 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:57.799712 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:57.800120 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:57.800489 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:57.800496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:57.800499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:57.800668 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:57.801155 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:57.801162 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:57.801472 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 01:07:57.801795 (+ 323us) client_negotiation.cc:240] Negotiation successful
0929 01:07:57.802287 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":298,"threads_started":1}
I20260929 01:07:57.805127 12513 heartbeater.cc:347] Connected to a master server at 127.5.198.254:39173
I20260929 01:07:57.805451 12513 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:57.806167 12513 heartbeater.cc:511] Master 127.5.198.254:39173 requested a full tablet report, sending...
I20260929 01:07:57.807516 12057 ts_manager.cc:194] Registered new tserver with Master: 59b3bc938cb34499a7aa7607b65ea443 (127.5.198.195:35401)
I20260929 01:07:57.808274 12057 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:42029
I20260929 01:07:57.818321 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:07:57.833523 12246 heartbeater.cc:503] Master 127.5.198.254:39173 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.5.198.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.198.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.198.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.253.keytab.
Entry for principal oryx/127.5.198.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.253.keytab.
I20260929 01:07:57.876317 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.253:44255
--webserver_interface=127.5.198.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.253.keytab
--principal=oryx/127.5.198.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:39173,127.5.198.253:44255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:58.224049 12518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:58.224308 12518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:58.224514 12518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:58.231679 12518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:58.231772 12518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:58.231895 12518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:58.231997 12518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:58.232082 12518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:07:58.241935 12518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:39173,127.5.198.253:44255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.253.keytab
--principal=oryx/127.5.198.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.253:44255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.198.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
W20260929 01:07:58.242301 12518 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.198.254:39173,127.5.198.253:44255'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 01:07:58.243386 12518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:58.244732 12518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:58.255578 12523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:58.256222 12524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:58.256661 12526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:58.257508 12518 server_base.cc:1034] running on GCE node
I20260929 01:07:58.258314 12518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:58.259648 12518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:58.260900 12518 hybrid_clock.cc:648] HybridClock initialized: now 1790644078260826 us; error 45 us; skew 500 ppm
I20260929 01:07:58.261219 12518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:58 dist-test-slave-41cl krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644078, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:58.264142 12518 init.cc:377] Logged in from keytab as oryx/127.5.198.253@KRBTEST.COM (short username oryx)
I20260929 01:07:58.265635 12518 webserver.cc:460] Webserver started at http://127.5.198.253:46407/ using document root <none> and password file <none>
I20260929 01:07:58.266300 12518 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:58.266440 12518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:58.266741 12518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:58.269609 12518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/data/instance:
uuid: "1bfef164acd2450381b461e655b0500a"
format_stamp: "Formatted at 2026-09-29 01:07:58 on dist-test-slave-41cl"
server_key: "70f3a705647afe84aac6108edb38c43f"
server_key_iv: "9b964b272c86f6ccb56830ea72157d3d"
server_key_version: "encryptionkey@0"
I20260929 01:07:58.270355 12518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/wal/instance:
uuid: "1bfef164acd2450381b461e655b0500a"
format_stamp: "Formatted at 2026-09-29 01:07:58 on dist-test-slave-41cl"
server_key: "70f3a705647afe84aac6108edb38c43f"
server_key_iv: "9b964b272c86f6ccb56830ea72157d3d"
server_key_version: "encryptionkey@0"
I20260929 01:07:58.275183 12518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:07:58.278683 12533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:58.280043 12518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 01:07:58.280229 12518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/wal
uuid: "1bfef164acd2450381b461e655b0500a"
format_stamp: "Formatted at 2026-09-29 01:07:58 on dist-test-slave-41cl"
server_key: "70f3a705647afe84aac6108edb38c43f"
server_key_iv: "9b964b272c86f6ccb56830ea72157d3d"
server_key_version: "encryptionkey@0"
I20260929 01:07:58.280438 12518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:58.292097 12518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:58.295771 12518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:58.296031 12518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:58.314261 12518 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.253:44255
I20260929 01:07:58.314308 12584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.253:44255 every 8 connection(s)
I20260929 01:07:58.315564 12518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/data/info.pb
I20260929 01:07:58.319205 12585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:58.319653 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 12518
I20260929 01:07:58.325088 12380 heartbeater.cc:503] Master 127.5.198.254:39173 was elected leader, sending a full tablet report...
I20260929 01:07:58.328266 12585 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:07:58 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644078, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.253@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM
Sep 29 01:07:58 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.198.253@KRBTEST.COM
I20260929 01:07:58.350643 12590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:58.331503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.5.198.254:39173)
0929 01:07:58.332742 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:58.332746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:58.332761 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:07:58.333629 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:58.333635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:58.333716 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:58.333830 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:58.335649 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.336859 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.339056 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.339715 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.339861 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:58.344132 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:58.344139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:58.344144 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:58.344182 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:58.347542 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:58.348272 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:58.348279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:58.348285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:58.348434 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:58.348889 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:58.348896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:58.348902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:58.349217 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:58.349367 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:58.350073 (+ 706us) server_negotiation.cc:299] Negotiation successful
0929 01:07:58.350281 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1149,"thread_start_us":1032,"threads_started":1}
I20260929 01:07:58.351766 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:58.331445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.0.0.1:48334)
0929 01:07:58.332139 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:58.332178 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:58.333390 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:58.334062 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:58.334070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:58.334881 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:58.335421 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:58.335446 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.337061 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.337068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:58.338908 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:58.338923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.339948 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.339953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:58.340203 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:58.341235 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:58.341273 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:58.343942 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:58.347668 (+ 3726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:58.347680 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:58.347693 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:58.348173 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:58.348584 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:58.348592 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:58.348596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:58.348776 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:58.349353 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:58.349361 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:58.349690 (+ 329us) client_negotiation.cc:769] Sending connection context
0929 01:07:58.349973 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 01:07:58.350514 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":1,"thread_start_us":317,"threads_started":1}
I20260929 01:07:58.351747 12588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:58.324756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.5.198.253:44255)
0929 01:07:58.331113 (+ 6357us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:58.331122 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:58.331158 (+ 36us) server_negotiation.cc:407] Connection header received
0929 01:07:58.333372 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:58.333393 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:58.333612 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:58.333870 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:58.335665 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.337350 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.339121 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.339944 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.340103 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:58.345506 (+ 5403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:58.345528 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:58.345546 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:58.345585 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:58.348784 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:58.349394 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:58.349402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:58.349405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:58.349532 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:58.349917 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:58.349924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:58.349927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:58.350244 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:58.350591 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:58.351130 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 01:07:58.351340 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":6172,"thread_start_us":5977,"threads_started":1}
I20260929 01:07:58.354295 12585 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:07:58 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644078, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.253@KRBTEST.COM for oryx/127.5.198.253@KRBTEST.COM
I20260929 01:07:58.365962 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:58.355030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:44255 (local address 127.0.0.1:46972)
0929 01:07:58.355199 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:58.355218 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:58.355304 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:58.355526 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:58.355529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:58.355808 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:58.355924 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:58.355931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.356944 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.356948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:58.357777 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:58.357786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.358446 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.358451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:58.358636 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:58.359249 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:58.359269 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:58.361251 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:58.363979 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:58.363982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:58.363985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:58.364326 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:58.364692 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:58.364695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:58.364697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:58.364824 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:58.365267 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:58.365272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:58.365388 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 01:07:58.365491 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 01:07:58.365677 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 01:07:58.366035 12588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:58.355158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46972 (local address 127.5.198.253:44255)
0929 01:07:58.355262 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:58.355266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:58.355279 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:07:58.355375 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:58.355378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:58.355424 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:58.355498 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:58.356032 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.356771 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.357910 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.358288 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.358381 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:58.361419 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:58.361424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:58.361427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:58.361452 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:58.363863 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:58.364418 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:58.364421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:58.364423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:58.364601 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:58.364913 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:58.364916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:58.364918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:58.365174 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:58.365275 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:58.365590 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 01:07:58.365781 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":28}
I20260929 01:07:58.370721 12585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: Bootstrap starting.
I20260929 01:07:58.373706 12585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:58.374683 12585 log.cc:824] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:58.377197 12585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: No bootstrap required, opened a new log
I20260929 01:07:58.383941 12585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } }
I20260929 01:07:58.384255 12585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:58.384383 12585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bfef164acd2450381b461e655b0500a, State: Initialized, Role: FOLLOWER
I20260929 01:07:58.384976 12585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } }
I20260929 01:07:58.385737 12592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } } }
I20260929 01:07:58.386034 12592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:07:58.386682 12585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:07:58.394315 12603 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:07:58.394446 12603 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:07:58.397042 12518 master_runner.cc:428] Detected that this master 1bfef164acd2450381b461e655b0500a is joining an existing cluster
I20260929 01:07:58.397143 12518 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.198.253:44255
I20260929 01:07:58.419288 12588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:58.403094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.5.198.253:44255)
0929 01:07:58.403271 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:58.403277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:58.404017 (+ 740us) server_negotiation.cc:407] Connection header received
0929 01:07:58.404166 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:58.404173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:58.404236 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:58.404330 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:58.405883 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.406958 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.409090 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.409805 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.409896 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:58.412630 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:58.412638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:58.412647 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:58.412679 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:58.416259 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:58.417076 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:58.417081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:58.417084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:58.417245 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:58.417692 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:58.417697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:58.417700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:58.418073 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:58.418235 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:58.418681 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 01:07:58.418901 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56,"spinlock_wait_cycles":2432}
I20260929 01:07:58.420258 12610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:58.402855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:44255 (local address 127.0.0.1:46982)
0929 01:07:58.403907 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:58.403925 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:58.404042 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:58.404375 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:58.404380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:58.405392 (+ 1012us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:58.405737 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:58.405749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.407182 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.407188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:58.408951 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:58.408965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.410127 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.410134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:58.410371 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:58.410984 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:58.411013 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:58.412343 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:58.416409 (+ 4066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:58.416414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:58.416418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:58.416974 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:58.417368 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:58.417373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:58.417377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:58.417566 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:58.418300 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:58.418304 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:58.418442 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 01:07:58.419524 (+ 1082us) client_negotiation.cc:240] Negotiation successful
0929 01:07:58.419792 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":102,"spinlock_wait_cycles":26368,"thread_start_us":757,"threads_started":1}
I20260929 01:07:58.420908 12590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:58.402713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.5.198.254:39173)
0929 01:07:58.402907 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:58.402916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:58.403574 (+ 658us) server_negotiation.cc:407] Connection header received
0929 01:07:58.403864 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:58.403869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:58.403938 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:58.404032 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:58.405542 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.406600 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.409136 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.409850 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.409957 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:58.412783 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:58.412791 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:58.412798 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:58.412841 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:58.416909 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:58.417676 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:58.417684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:58.417690 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:58.417829 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:58.418276 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:58.418283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:58.418290 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:58.418651 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:58.418788 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:58.419682 (+ 894us) server_negotiation.cc:299] Negotiation successful
0929 01:07:58.419939 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:07:58.422422 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:58.402728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.0.0.1:48350)
0929 01:07:58.403448 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:58.403474 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:58.403696 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:58.404241 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:58.404245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:58.404947 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:58.405366 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:58.405378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.406782 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.406789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:58.408981 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:58.408998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.410273 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.410282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:58.410526 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:58.411202 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:58.411235 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:58.412589 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:58.417072 (+ 4483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:58.417077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:58.417081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:58.417550 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:58.417966 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:58.417970 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:58.417976 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:58.418151 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:58.418867 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:58.418874 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:58.419085 (+ 211us) client_negotiation.cc:769] Sending connection context
0929 01:07:58.421391 (+ 2306us) client_negotiation.cc:240] Negotiation successful
0929 01:07:58.421761 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":55,"spinlock_wait_cycles":30848,"thread_start_us":538,"threads_started":1}
WARNING: no policy specified for oryx/127.5.198.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.198.252@KRBTEST.COM" created.
Sep 29 01:07:58 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644076, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.254@KRBTEST.COM for oryx/127.5.198.253@KRBTEST.COM
I20260929 01:07:58.448622 12588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:58.429063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.5.198.253:44255)
0929 01:07:58.429225 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:58.429231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:58.433180 (+ 3949us) server_negotiation.cc:407] Connection header received
0929 01:07:58.433533 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:58.433539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:58.433648 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:58.433758 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:58.434878 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.435570 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.436949 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.437411 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.437562 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:58.441185 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:58.441192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:58.441197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:58.441237 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:58.444987 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:58.445956 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:58.445962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:58.445965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:58.446140 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:58.446615 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:58.446623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:58.446629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:58.446999 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:58.447166 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:58.447743 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 01:07:58.448035 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:07:58.448635 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:58.428963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:44255 (local address 127.0.0.1:46988)
0929 01:07:58.433045 (+ 4082us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:58.433083 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:58.433308 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:58.433803 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:58.433813 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:58.434512 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:58.434719 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:58.434728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.435701 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.435708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:58.436806 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:58.436818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.437580 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.437586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:58.437770 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:58.438476 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:58.438537 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:58.440982 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:58.445138 (+ 4156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:58.445153 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:58.445166 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:58.445779 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:58.446272 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:58.446280 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:58.446286 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:58.446465 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:58.447138 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:58.447149 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:58.447385 (+ 236us) client_negotiation.cc:769] Sending connection context
0929 01:07:58.447610 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 01:07:58.448016 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3825,"thread_start_us":3680,"threads_started":1}
I20260929 01:07:58.452105 12057 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.198.253:44255: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1bfef164acd2450381b461e655b0500a"
member_type: NON_VOTER
last_known_addr {
host: "127.5.198.253"
port: 44255
}
attrs {
promote: true
}
}
dest_uuid: "03087ec95dd74fec8f8ccd4af29de874"
cas_config_opid_index: -1
I20260929 01:07:58.453397 12057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: NON_VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: true } }
I20260929 01:07:58.456806 12616 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1bfef164acd2450381b461e655b0500a (127.5.198.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: NON_VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: true } } }
I20260929 01:07:58.457818 12560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:58.459976 12616 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.198.253:44255
I20260929 01:07:58.460361 12615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "03087ec95dd74fec8f8ccd4af29de874" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: NON_VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: true } } }
I20260929 01:07:58.460816 12615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:58.461314 12518 master.cc:537] Master@127.5.198.253:44255 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.198.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.252.keytab.
Entry for principal oryx/127.5.198.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.252.keytab.
I20260929 01:07:58.465457 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.252:39101
--webserver_interface=127.5.198.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.252.keytab
--principal=oryx/127.5.198.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:39173,127.5.198.253:44255,127.5.198.252:39101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:07:58.465646 12560 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 1 FOLLOWER]: Refusing update from remote peer 03087ec95dd74fec8f8ccd4af29de874: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:07:58.468658 12615 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: NON_VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 01:07:58.471354 12043 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 01:07:58.474756 12043 consensus_peers.cc:487] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 -> Peer 1bfef164acd2450381b461e655b0500a (127.5.198.253:44255): Couldn't send request to peer 1bfef164acd2450381b461e655b0500a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 01:07:58.476051 12518 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:07:58.476452 12518 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:58.476681 12518 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: stopping tablet replica
I20260929 01:07:58.483203 12518 master.cc:552] Master@127.5.198.253:44255 shutdown complete.
I20260929 01:07:58.483312 12518 master_runner.cc:305] Clearing existing system tablet
I20260929 01:07:58.483666 12518 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:58.483770 12518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:58.486147 12518 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:07:58.487785 12621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:58.488036 12518 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:58.488319 12518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 01:07:58.488399 12518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/wal
uuid: "1bfef164acd2450381b461e655b0500a"
format_stamp: "Formatted at 2026-09-29 01:07:58 on dist-test-slave-41cl"
server_key: "70f3a705647afe84aac6108edb38c43f"
server_key_iv: "9b964b272c86f6ccb56830ea72157d3d"
server_key_version: "encryptionkey@0"
I20260929 01:07:58.491909 12518 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:58.496412 12518 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:07:58.496649 12518 log.cc:1196] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 01:07:58.497009 12518 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: Deleting consensus metadata
I20260929 01:07:58.497268 12518 master_runner.cc:315] Copying system tablet from 127.5.198.254:39173
I20260929 01:07:58.501709 12518 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.198.254:39173
I20260929 01:07:58.515959 12590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:58.502464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.5.198.254:39173)
0929 01:07:58.502709 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:58.502718 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:58.504989 (+ 2271us) server_negotiation.cc:407] Connection header received
0929 01:07:58.505202 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:58.505207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:58.505282 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:58.505390 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:58.506239 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.506939 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.508161 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.508663 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.508815 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:58.510547 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:58.510553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:58.510557 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:58.510594 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:58.513436 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:58.514028 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:58.514035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:58.514039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:58.514178 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:58.514538 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:58.514544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:58.514546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:58.514825 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:58.514982 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:58.515378 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 01:07:58.515567 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:07:58.516027 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:58.502389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.0.0.1:48358)
0929 01:07:58.504872 (+ 2483us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:58.504900 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:58.505056 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:58.505410 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:58.505415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:58.505822 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:58.506013 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:58.506022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.507088 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.507094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:58.508006 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:58.508017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.508828 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.508834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:58.509013 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:58.509655 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:58.509671 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:58.510322 (+ 651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:58.513560 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:58.513567 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:58.513571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:58.513910 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:58.514296 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:58.514300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:58.514304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:58.514433 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:58.514941 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:58.514946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:58.515104 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 01:07:58.515269 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 01:07:58.515551 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2352,"thread_start_us":2273,"threads_started":1}
I20260929 01:07:58.517179 12077 tablet_copy_service.cc:140] P 03087ec95dd74fec8f8ccd4af29de874: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1bfef164acd2450381b461e655b0500a ({username='oryx', principal='oryx/127.5.198.253@KRBTEST.COM'} at 127.0.0.1:48358)
I20260929 01:07:58.517427 12077 tablet_copy_service.cc:161] P 03087ec95dd74fec8f8ccd4af29de874: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1bfef164acd2450381b461e655b0500a at {username='oryx', principal='oryx/127.5.198.253@KRBTEST.COM'} at 127.0.0.1:48358: session id = 1bfef164acd2450381b461e655b0500a-00000000000000000000000000000000
I20260929 01:07:58.520704 12077 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:07:58.522804 12518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:07:58.527582 12518 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: tablet copy: Starting download of 0 data blocks...
I20260929 01:07:58.527865 12518 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: tablet copy: Starting download of 1 WAL segments...
I20260929 01:07:58.530416 12518 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:07:58.533512 12077 tablet_copy_service.cc:342] P 03087ec95dd74fec8f8ccd4af29de874: Request end of tablet copy session 1bfef164acd2450381b461e655b0500a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.198.253@KRBTEST.COM'} at 127.0.0.1:48358
I20260929 01:07:58.533699 12077 tablet_copy_service.cc:433] P 03087ec95dd74fec8f8ccd4af29de874: ending tablet copy session 1bfef164acd2450381b461e655b0500a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1bfef164acd2450381b461e655b0500a
I20260929 01:07:58.544019 12518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:58.546602 12628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:58.556818 12518 server_base.cc:1034] running on GCE node
W20260929 01:07:58.557161 12629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:58.557606 12631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:58.557977 12518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:58.558578 12518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:58.559808 12518 hybrid_clock.cc:648] HybridClock initialized: now 1790644078559746 us; error 35 us; skew 500 ppm
I20260929 01:07:58.560055 12518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:58 dist-test-slave-41cl krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644078, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:58.561973 12518 init.cc:377] Logged in from keytab as oryx/127.5.198.253@KRBTEST.COM (short username oryx)
I20260929 01:07:58.563176 12518 webserver.cc:460] Webserver started at http://127.5.198.253:40329/ using document root <none> and password file <none>
I20260929 01:07:58.563486 12518 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:58.563584 12518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:58.565902 12518 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:07:58.567488 12638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:58.567914 12518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 01:07:58.568061 12518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/wal
uuid: "1bfef164acd2450381b461e655b0500a"
format_stamp: "Formatted at 2026-09-29 01:07:58 on dist-test-slave-41cl"
server_key: "70f3a705647afe84aac6108edb38c43f"
server_key_iv: "9b964b272c86f6ccb56830ea72157d3d"
server_key_version: "encryptionkey@0"
I20260929 01:07:58.568214 12518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:58.589859 12518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:58.593623 12518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:58.620496 12518 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.253:44255
I20260929 01:07:58.620447 12689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.253:44255 every 8 connection(s)
I20260929 01:07:58.621470 12518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-1/data/info.pb
I20260929 01:07:58.631413 12690 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:07:58.636883 12690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: Bootstrap starting.
I20260929 01:07:58.649982 12690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:07:58.650430 12690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: Bootstrap complete.
I20260929 01:07:58.651338 12690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: NON_VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: true } }
I20260929 01:07:58.651618 12690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1bfef164acd2450381b461e655b0500a, State: Initialized, Role: LEARNER
I20260929 01:07:58.651839 12690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: NON_VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: true } }
I20260929 01:07:58.652637 12693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: NON_VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: true } } }
I20260929 01:07:58.652971 12690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:58.653069 12693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: This master's current role is: LEARNER
I20260929 01:07:58.662530 12704 catalog_manager.cc:1259] Loaded cluster ID: 55239c9c72aa4bf08cacc386bfc02c32
I20260929 01:07:58.662710 12704 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: loading cluster ID for follower catalog manager: success
I20260929 01:07:58.665081 12704 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: acquiring CA information for follower catalog manager: success
I20260929 01:07:58.666715 12704 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:07:58.810778 12513 heartbeater.cc:503] Master 127.5.198.254:39173 was elected leader, sending a full tablet report...
W20260929 01:07:58.867679 12619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:58.868021 12619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:58.868175 12619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:58.875694 12619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:07:58.875835 12619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:58.875937 12619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:58.876034 12619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:07:58.876097 12619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:07:58.886030 12619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:39173,127.5.198.253:44255,127.5.198.252:39101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.252.keytab
--principal=oryx/127.5.198.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.252:39101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.198.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:58.887310 12619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:58.888574 12619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:58.900588 12709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:58.900638 12710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:58.900769 12712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:58.901422 12619 server_base.cc:1034] running on GCE node
I20260929 01:07:58.902123 12619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:58.903512 12619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:58.904762 12619 hybrid_clock.cc:648] HybridClock initialized: now 1790644078904695 us; error 48 us; skew 500 ppm
I20260929 01:07:58.905123 12619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:58 dist-test-slave-41cl krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644078, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:58.908129 12619 init.cc:377] Logged in from keytab as oryx/127.5.198.252@KRBTEST.COM (short username oryx)
I20260929 01:07:58.909672 12619 webserver.cc:460] Webserver started at http://127.5.198.252:45555/ using document root <none> and password file <none>
I20260929 01:07:58.910348 12619 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:58.910493 12619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:58.910789 12619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:07:58.913700 12619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/data/instance:
uuid: "9ecc84047a304e718e1f8e6eb02f0a74"
format_stamp: "Formatted at 2026-09-29 01:07:58 on dist-test-slave-41cl"
server_key: "29c0f4e7a85ebb10c5a2f62343537f05"
server_key_iv: "47553f66fe407350f5d487270ebc85ac"
server_key_version: "encryptionkey@0"
I20260929 01:07:58.914440 12619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/wal/instance:
uuid: "9ecc84047a304e718e1f8e6eb02f0a74"
format_stamp: "Formatted at 2026-09-29 01:07:58 on dist-test-slave-41cl"
server_key: "29c0f4e7a85ebb10c5a2f62343537f05"
server_key_iv: "47553f66fe407350f5d487270ebc85ac"
server_key_version: "encryptionkey@0"
I20260929 01:07:58.919752 12619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:07:58.923229 12719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:58.924598 12619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:07:58.924768 12619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/wal
uuid: "9ecc84047a304e718e1f8e6eb02f0a74"
format_stamp: "Formatted at 2026-09-29 01:07:58 on dist-test-slave-41cl"
server_key: "29c0f4e7a85ebb10c5a2f62343537f05"
server_key_iv: "47553f66fe407350f5d487270ebc85ac"
server_key_version: "encryptionkey@0"
I20260929 01:07:58.924961 12619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:58.945411 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:58.939461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:44255 (local address 127.0.0.1:47002)
0929 01:07:58.939831 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:58.939857 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:58.939996 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:58.940548 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:58.940554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:58.940583 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:58.940817 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:58.940826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.942178 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.942183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:58.943425 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:58.943437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.944592 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.944598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:58.944813 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:58.944877 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 01:07:58.944990 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 01:07:58.945119 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260929 01:07:58.945437 12724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:58.939679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.5.198.253:44255)
0929 01:07:58.940215 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:58.940219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:58.940238 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:07:58.940294 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:58.940298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:58.940362 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:58.940515 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:58.940960 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.942014 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.943575 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:58.944393 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:58.944562 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:58.944809 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:58.945018 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 01:07:58.945139 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":272,"threads_started":1}
I20260929 01:07:58.946460 12665 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 01:07:58.947116 12693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03087ec95dd74fec8f8ccd4af29de874. Latest consensus state: current_term: 1 leader_uuid: "03087ec95dd74fec8f8ccd4af29de874" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: NON_VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: true } } }
I20260929 01:07:58.947356 12693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: This master's current role is: LEARNER
I20260929 01:07:58.963790 12619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:58.967569 12619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:58.967793 12619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:58.986204 12619 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.252:39101
I20260929 01:07:58.986234 12771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.252:39101 every 8 connection(s)
I20260929 01:07:58.987921 12619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/data/info.pb
I20260929 01:07:58.991613 12772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:07:58.994503 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 12619
I20260929 01:07:58.999866 12772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:07:59 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.198.252@KRBTEST.COM
Sep 29 01:07:59 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644078, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.252@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM
I20260929 01:07:59.019222 12590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.002416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.5.198.254:39173)
0929 01:07:59.002645 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:59.002654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:59.003270 (+ 616us) server_negotiation.cc:407] Connection header received
0929 01:07:59.003418 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:59.003422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:59.003482 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:59.003564 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:59.004855 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.005787 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.007548 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.008086 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.008206 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.012356 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:59.012362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:59.012367 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:59.012408 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:59.016158 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.017022 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.017027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.017031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.017181 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.017617 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.017622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.017625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.017943 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:59.018092 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:59.018655 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 01:07:59.018888 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 01:07:59.020202 12775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:58.996790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39418 (local address 127.5.198.252:39101)
0929 01:07:58.997652 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:58.997664 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:58.997710 (+ 46us) server_negotiation.cc:407] Connection header received
0929 01:07:58.999332 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:58.999353 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:58.999779 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:59.000251 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:59.002070 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.004014 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.005638 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.006640 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.006827 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.011988 (+ 5161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:59.012000 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:59.012012 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:59.012044 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:59.016104 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.017022 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.017030 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.017033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.017188 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.017617 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.017622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.017625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.018269 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:59.018531 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:59.018891 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 01:07:59.019187 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":666,"spinlock_wait_cycles":14720,"thread_start_us":377,"threads_started":1}
I20260929 01:07:59.020244 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.002338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.0.0.1:48362)
0929 01:07:59.003116 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:59.003164 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:59.003305 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:59.003652 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:59.003659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:59.004401 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:59.004705 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.004713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.005945 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.005950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.007384 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.007393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.008338 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.008346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.008588 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.009316 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:59.009349 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:59.012152 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:59.016275 (+ 4123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.016285 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.016297 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.016885 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.017304 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.017308 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.017312 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.017503 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.018061 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:59.018067 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:59.018238 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 01:07:59.018541 (+ 303us) client_negotiation.cc:240] Negotiation successful
0929 01:07:59.019082 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":28,"thread_start_us":362,"threads_started":1}
I20260929 01:07:59.021703 12772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:07:59 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644078, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.252@KRBTEST.COM for oryx/127.5.198.253@KRBTEST.COM
I20260929 01:07:59.033378 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.022476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:44255 (local address 127.0.0.1:47016)
0929 01:07:59.022691 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:59.022704 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:59.022838 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:59.023210 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:59.023213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:59.023488 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:59.023606 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.023613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.024585 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.024588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.025442 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.025451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.026042 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.026046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.026162 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.026602 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:59.026617 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:59.028459 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:59.031474 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.031477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.031479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.031800 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.032107 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.032110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.032112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.032218 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.032721 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:59.032724 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:59.032823 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 01:07:59.032925 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 01:07:59.033104 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260929 01:07:59.033401 12724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.022437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.5.198.253:44255)
0929 01:07:59.022595 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:59.022600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:59.022780 (+ 180us) server_negotiation.cc:407] Connection header received
0929 01:07:59.022988 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:59.022994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:59.023064 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:59.023198 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:59.023749 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.024435 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.025553 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.025931 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.026021 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.028680 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:59.028688 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:59.028695 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:59.028733 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:59.031360 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.031895 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.031899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.031901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.032017 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.032304 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.032307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.032309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.032628 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:59.032734 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:59.033008 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 01:07:59.033159 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:07:59.034336 12772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:07:59 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644078, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.252@KRBTEST.COM for oryx/127.5.198.252@KRBTEST.COM
I20260929 01:07:59.046464 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.034926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:39101 (local address 127.0.0.1:39422)
0929 01:07:59.035131 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:59.035149 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:59.035231 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:59.035868 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:59.035871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:59.036161 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:59.036282 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.036289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.037208 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.037212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.038131 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.038145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.038833 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.038837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.038986 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.039396 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:59.039410 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:59.041389 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:59.044378 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.044382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.044384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.044816 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.045127 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.045130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.045132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.045279 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.045704 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:59.045707 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:59.045820 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 01:07:59.045939 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 01:07:59.046107 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 01:07:59.046471 12775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.035516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.5.198.252:39101)
0929 01:07:59.035658 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:59.035662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:59.035676 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:07:59.035709 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:59.035712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:59.035758 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:59.035833 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:59.036391 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.037108 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.038258 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.038712 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.038814 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.041550 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:59.041555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:59.041558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:59.041583 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:59.044255 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.044918 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.044922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.044924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.045036 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.045368 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.045371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.045373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.045606 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:59.045704 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:59.046022 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 01:07:59.046204 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":56}
I20260929 01:07:59.051049 12772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: Bootstrap starting.
I20260929 01:07:59.054029 12772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: Neither blocks nor log segments found. Creating new log.
I20260929 01:07:59.055397 12772 log.cc:824] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: Log is configured to *not* fsync() on all Append() calls
I20260929 01:07:59.057901 12772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: No bootstrap required, opened a new log
I20260929 01:07:59.064621 12772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } }
I20260929 01:07:59.064941 12772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:07:59.065052 12772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecc84047a304e718e1f8e6eb02f0a74, State: Initialized, Role: FOLLOWER
I20260929 01:07:59.065630 12772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } }
I20260929 01:07:59.066460 12778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } } }
I20260929 01:07:59.066787 12778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:07:59.067371 12772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:59.075145 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 12118
W20260929 01:07:59.076579 12789 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:07:59.076697 12789 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:07:59.080713 12619 master_runner.cc:428] Detected that this master 9ecc84047a304e718e1f8e6eb02f0a74 is joining an existing cluster
I20260929 01:07:59.080823 12619 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.198.252:39101
Sep 29 01:07:59 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.198.193@KRBTEST.COM
I20260929 01:07:59.103922 12590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.087542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48372 (local address 127.5.198.254:39173)
0929 01:07:59.087944 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:59.087953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:59.088735 (+ 782us) server_negotiation.cc:407] Connection header received
0929 01:07:59.088931 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:59.088936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:59.089009 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:59.089109 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:59.090002 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.090895 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.092908 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.093504 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.093592 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.096740 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:59.096749 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:59.096755 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:59.096798 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:59.100202 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.100978 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.100985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.100990 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.101158 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.101623 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.101628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.101632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.102005 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:59.102148 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:59.102843 (+ 695us) server_negotiation.cc:299] Negotiation successful
0929 01:07:59.103084 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260929 01:07:59.104894 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.086764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.0.0.1:48372)
0929 01:07:59.088621 (+ 1857us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:59.088639 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:59.088789 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:59.089128 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:59.089132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:59.089621 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:59.089857 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.089865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.091062 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.091070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.092743 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.092756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.093859 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.093864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.094162 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.095393 (+ 1231us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:59.095415 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:59.096536 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:59.100341 (+ 3805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.100345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.100349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.100818 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.101313 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.101318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.101321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.101495 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.102246 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:59.102250 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:59.102500 (+ 250us) client_negotiation.cc:769] Sending connection context
0929 01:07:59.104109 (+ 1609us) client_negotiation.cc:240] Negotiation successful
0929 01:07:59.104407 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1723,"mutex_wait_us":25,"threads_started":1}
I20260929 01:07:59.105449 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.076775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.5.198.193:35399)
0929 01:07:59.077941 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:59.077951 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:59.077980 (+ 29us) server_negotiation.cc:407] Connection header received
0929 01:07:59.078048 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:59.078056 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:59.078300 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:59.078458 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:59.080440 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.082156 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.083982 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.084908 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.085042 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.095462 (+ 10420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:59.095477 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:59.095494 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:59.095541 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:59.099641 (+ 4100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.100541 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.100554 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.100560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.100739 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.101306 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.101314 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.101319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.101836 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:59.103591 (+ 1755us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:59.103789 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 01:07:59.104608 (+ 819us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":985,"thread_start_us":829,"threads_started":1}
I20260929 01:07:59.107409 12241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:59.107751 12724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.087322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47028 (local address 127.5.198.253:44255)
0929 01:07:59.087776 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:59.087784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:59.091107 (+ 3323us) server_negotiation.cc:407] Connection header received
0929 01:07:59.091369 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:59.091376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:59.091450 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:59.091514 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:59.092871 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.093865 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.096801 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.097462 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.097576 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.100175 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:59.100182 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:59.100186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:59.100220 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:59.103987 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.104820 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.104827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.104832 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.104982 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.105452 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.105459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.105466 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.105787 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:59.105914 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:59.106541 (+ 627us) server_negotiation.cc:299] Negotiation successful
0929 01:07:59.106756 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299}
I20260929 01:07:59.109417 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.087204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:44255 (local address 127.0.0.1:47028)
0929 01:07:59.091027 (+ 3823us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:59.091044 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:59.091202 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:59.091701 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:59.091706 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:59.092337 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:59.092697 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.092709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.094053 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.094057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.096667 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.096679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.097864 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.097873 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.098076 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.098884 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:59.098907 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:59.099974 (+ 1067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:59.104219 (+ 4245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.104223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.104227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.104690 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.105131 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.105138 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.105144 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.105329 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.106001 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:59.106009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:59.106222 (+ 213us) client_negotiation.cc:769] Sending connection context
0929 01:07:59.108634 (+ 2412us) client_negotiation.cc:240] Negotiation successful
0929 01:07:59.108938 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3679,"mutex_wait_us":40,"thread_start_us":1099,"threads_started":1}
I20260929 01:07:59.120792 12775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.090109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.5.198.252:39101)
0929 01:07:59.090779 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:59.090788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:59.105159 (+ 14371us) server_negotiation.cc:407] Connection header received
0929 01:07:59.105345 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:59.105349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:59.105420 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:59.105480 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:59.106499 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.107493 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.109485 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.110228 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.110327 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.112438 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:59.112444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:59.112448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:59.112503 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:59.116762 (+ 4259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.117712 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.117720 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.117726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.117876 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.118726 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.118733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.118739 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.119178 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:59.119315 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:59.119995 (+ 680us) server_negotiation.cc:299] Negotiation successful
0929 01:07:59.120274 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528}
I20260929 01:07:59.122576 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.090388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:39101 (local address 127.0.0.1:39436)
0929 01:07:59.105094 (+ 14706us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:59.105112 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:59.105234 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:59.105648 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:59.105653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:59.106180 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:59.106372 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.106385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.107713 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.107717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.109296 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.109312 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.110717 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.110726 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.110907 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.111466 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:59.111487 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:59.112284 (+ 797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:59.116930 (+ 4646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.116938 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.116944 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.117561 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.118027 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.118033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.118039 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.118605 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.119424 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:59.119431 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:59.119658 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 01:07:59.121641 (+ 1983us) client_negotiation.cc:240] Negotiation successful
0929 01:07:59.121992 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14598,"mutex_wait_us":3,"thread_start_us":2394,"threads_started":1}
Sep 29 01:07:59 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644076, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.254@KRBTEST.COM for oryx/127.5.198.252@KRBTEST.COM
I20260929 01:07:59.135983 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.113607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:39101 (local address 127.0.0.1:39450)
0929 01:07:59.113836 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:59.113853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:59.113948 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:59.121488 (+ 7540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:59.121494 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:59.121929 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:59.122105 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.122116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.123537 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.123544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.124943 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.124958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.125978 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.125988 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.126237 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.126948 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:59.126979 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:59.130243 (+ 3264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:59.133768 (+ 3525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.133773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.133778 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.134100 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.134458 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.134463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.134467 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.134590 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.135069 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:59.135074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:59.135219 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 01:07:59.135372 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 01:07:59.135616 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260929 01:07:59.136544 12775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.113674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39450 (local address 127.5.198.252:39101)
0929 01:07:59.121063 (+ 7389us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:59.121070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:59.121085 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:07:59.121170 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:59.121178 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:59.121315 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:59.121452 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:59.122275 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.123369 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.125127 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.125801 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.125925 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.130456 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:59.130462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:59.130468 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:59.130505 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:59.133602 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.134215 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.134222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.134225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.134348 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.134694 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.134703 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.134707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.134952 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:59.135096 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:59.135454 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 01:07:59.136243 (+ 789us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7300,"thread_start_us":15602,"threads_started":1}
I20260929 01:07:59.137876 12057 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.198.252:39101: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74"
member_type: NON_VOTER
last_known_addr {
host: "127.5.198.252"
port: 39101
}
attrs {
promote: true
}
}
dest_uuid: "03087ec95dd74fec8f8ccd4af29de874"
cas_config_opid_index: 5
I20260929 01:07:59.138825 12057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: NON_VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: true } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true } }
I20260929 01:07:59.141523 12665 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 1 LEARNER]: Refusing update from remote peer 03087ec95dd74fec8f8ccd4af29de874: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:07:59.142853 12615 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9ecc84047a304e718e1f8e6eb02f0a74 (127.5.198.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: NON_VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: true } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true } } }
I20260929 01:07:59.150262 12747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:07:59.153610 12747 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 1 FOLLOWER]: Refusing update from remote peer 03087ec95dd74fec8f8ccd4af29de874: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:07:59.156780 12615 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.198.252:39101
I20260929 01:07:59.156915 12802 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: NON_VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:07:59.157779 12615 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:07:59.157841 12801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "03087ec95dd74fec8f8ccd4af29de874" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: NON_VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: true } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true } } }
I20260929 01:07:59.158351 12801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:59.166651 12619 master.cc:537] Master@127.5.198.252:39101 shutting down...
I20260929 01:07:59.169075 12665 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9ecc84047a304e718e1f8e6eb02f0a74 (127.5.198.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: NON_VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: true } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true } } }
W20260929 01:07:59.173704 12042 consensus_peers.cc:487] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 -> Peer 9ecc84047a304e718e1f8e6eb02f0a74 (127.5.198.252:39101): Couldn't send request to peer 9ecc84047a304e718e1f8e6eb02f0a74. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 01:07:59.174889 12693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03087ec95dd74fec8f8ccd4af29de874" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: NON_VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: true } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true } } }
I20260929 01:07:59.175182 12693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: This master's current role is: LEARNER
I20260929 01:07:59.176071 12615 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874: attempting to promote NON_VOTER 1bfef164acd2450381b461e655b0500a to VOTER
I20260929 01:07:59.176983 12615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true } }
I20260929 01:07:59.179549 12664 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 1 LEARNER]: Refusing update from remote peer 03087ec95dd74fec8f8ccd4af29de874: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 01:07:59.180158 12802 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260929 01:07:59.180133 12042 consensus_peers.cc:487] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 -> Peer 9ecc84047a304e718e1f8e6eb02f0a74 (127.5.198.252:39101): Couldn't send request to peer 9ecc84047a304e718e1f8e6eb02f0a74. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 01:07:59.183089 12802 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1bfef164acd2450381b461e655b0500a (127.5.198.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true } } }
I20260929 01:07:59.186403 12801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "03087ec95dd74fec8f8ccd4af29de874" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true } } }
I20260929 01:07:59.186610 12801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:59.186376 12664 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1bfef164acd2450381b461e655b0500a (127.5.198.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true } } }
I20260929 01:07:59.186882 12801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03087ec95dd74fec8f8ccd4af29de874" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true } } }
I20260929 01:07:59.187088 12801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: This master's current role is: LEADER
I20260929 01:07:59.189459 12693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03087ec95dd74fec8f8ccd4af29de874" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true } } }
I20260929 01:07:59.189667 12693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:07:59.214551 12619 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:07:59.214927 12619 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:07:59.215147 12619 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: stopping tablet replica
I20260929 01:07:59.221122 12619 master.cc:552] Master@127.5.198.252:39101 shutdown complete.
I20260929 01:07:59.221215 12619 master_runner.cc:305] Clearing existing system tablet
I20260929 01:07:59.221544 12619 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:59.221663 12619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:59.223727 12619 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 01:07:59.225137 12808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:59.225399 12619 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:59.225667 12619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 01:07:59.225761 12619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/wal
uuid: "9ecc84047a304e718e1f8e6eb02f0a74"
format_stamp: "Formatted at 2026-09-29 01:07:58 on dist-test-slave-41cl"
server_key: "29c0f4e7a85ebb10c5a2f62343537f05"
server_key_iv: "47553f66fe407350f5d487270ebc85ac"
server_key_version: "encryptionkey@0"
I20260929 01:07:59.229038 12619 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:07:59.234213 12619 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:07:59.234372 12619 log.cc:1196] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 01:07:59.234759 12619 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: Deleting consensus metadata
I20260929 01:07:59.234997 12619 master_runner.cc:315] Copying system tablet from 127.5.198.254:39173
I20260929 01:07:59.237552 12619 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: tablet copy: Beginning tablet copy session from remote peer at address 127.5.198.254:39173
I20260929 01:07:59.250707 12814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.238319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.0.0.1:48386)
0929 01:07:59.238951 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:59.238969 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:59.239120 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:59.239467 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:59.239471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:59.239872 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:59.240089 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.240104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.241227 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.241230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.242341 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.242352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.243165 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.243171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.243365 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.244029 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:59.244045 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:59.244797 (+ 752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:59.248088 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.248093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.248097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.248664 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.249031 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.249035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.249039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.249175 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.249649 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:59.249654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:59.249821 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 01:07:59.249962 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 01:07:59.250252 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":370,"threads_started":1}
I20260929 01:07:59.251078 12590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.238319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48386 (local address 127.5.198.254:39173)
0929 01:07:59.238566 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:59.238573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:59.239057 (+ 484us) server_negotiation.cc:407] Connection header received
0929 01:07:59.239254 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:59.239260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:59.239336 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:59.239438 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:59.240264 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.241075 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.242495 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.243004 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.243153 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.244997 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:59.245003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:59.245007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:59.245043 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:59.247968 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.248773 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.248780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.248784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.248911 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.249276 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.249282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.249286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.249534 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:59.249694 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:59.250060 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 01:07:59.250258 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 01:07:59.251535 12077 tablet_copy_service.cc:140] P 03087ec95dd74fec8f8ccd4af29de874: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9ecc84047a304e718e1f8e6eb02f0a74 ({username='oryx', principal='oryx/127.5.198.252@KRBTEST.COM'} at 127.0.0.1:48386)
I20260929 01:07:59.251701 12077 tablet_copy_service.cc:161] P 03087ec95dd74fec8f8ccd4af29de874: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9ecc84047a304e718e1f8e6eb02f0a74 at {username='oryx', principal='oryx/127.5.198.252@KRBTEST.COM'} at 127.0.0.1:48386: session id = 9ecc84047a304e718e1f8e6eb02f0a74-00000000000000000000000000000000
I20260929 01:07:59.254407 12077 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:07:59.255873 12619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:07:59.260699 12619 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: tablet copy: Starting download of 0 data blocks...
I20260929 01:07:59.260972 12619 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: tablet copy: Starting download of 1 WAL segments...
I20260929 01:07:59.263091 12619 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:07:59.267023 12077 tablet_copy_service.cc:342] P 03087ec95dd74fec8f8ccd4af29de874: Request end of tablet copy session 9ecc84047a304e718e1f8e6eb02f0a74-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.198.252@KRBTEST.COM'} at 127.0.0.1:48386
I20260929 01:07:59.267151 12077 tablet_copy_service.cc:433] P 03087ec95dd74fec8f8ccd4af29de874: ending tablet copy session 9ecc84047a304e718e1f8e6eb02f0a74-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9ecc84047a304e718e1f8e6eb02f0a74
I20260929 01:07:59.273459 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 12118
I20260929 01:07:59.277660 12619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:59.280179 12815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:59.297237 12816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:59.298182 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:35399
--local_ip_for_outbound_sockets=127.5.198.193
--tserver_master_addrs=127.5.198.254:39173
--webserver_port=37325
--webserver_interface=127.5.198.193
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.193.keytab
--principal=oryx/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.198.254:39173,127.5.198.253:44255
--tserver_master_addrs=127.5.198.254:39173,127.5.198.253:44255,127.5.198.252:39101 with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:07:59.298511 12818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:59.299664 12619 server_base.cc:1034] running on GCE node
I20260929 01:07:59.300009 12619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:59.300657 12619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:59.301848 12619 hybrid_clock.cc:648] HybridClock initialized: now 1790644079301801 us; error 40 us; skew 500 ppm
I20260929 01:07:59.302083 12619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:59 dist-test-slave-41cl krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644079, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:59.304227 12619 init.cc:377] Logged in from keytab as oryx/127.5.198.252@KRBTEST.COM (short username oryx)
I20260929 01:07:59.305711 12619 webserver.cc:460] Webserver started at http://127.5.198.252:46793/ using document root <none> and password file <none>
I20260929 01:07:59.306078 12619 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:59.306171 12619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:59.309150 12619 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260929 01:07:59.311007 12826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:59.311506 12619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260929 01:07:59.311664 12619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/wal
uuid: "9ecc84047a304e718e1f8e6eb02f0a74"
format_stamp: "Formatted at 2026-09-29 01:07:58 on dist-test-slave-41cl"
server_key: "29c0f4e7a85ebb10c5a2f62343537f05"
server_key_iv: "47553f66fe407350f5d487270ebc85ac"
server_key_version: "encryptionkey@0"
I20260929 01:07:59.311905 12619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:59.331729 12619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:59.335608 12619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:59.374845 12619 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.252:39101
I20260929 01:07:59.374903 12877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.252:39101 every 8 connection(s)
I20260929 01:07:59.375998 12619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/master-2/data/info.pb
I20260929 01:07:59.381057 12878 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:07:59.384984 12878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: Bootstrap starting.
I20260929 01:07:59.407420 12878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:07:59.407899 12878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: Bootstrap complete.
I20260929 01:07:59.408965 12878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true } }
I20260929 01:07:59.409265 12878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9ecc84047a304e718e1f8e6eb02f0a74, State: Initialized, Role: LEARNER
I20260929 01:07:59.409548 12878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true } }
I20260929 01:07:59.410432 12881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true } } }
I20260929 01:07:59.410698 12878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:07:59.410859 12881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:07:59.420221 12892 catalog_manager.cc:1259] Loaded cluster ID: 55239c9c72aa4bf08cacc386bfc02c32
I20260929 01:07:59.420379 12892 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: loading cluster ID for follower catalog manager: success
I20260929 01:07:59.423668 12892 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: acquiring CA information for follower catalog manager: success
I20260929 01:07:59.425904 12892 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:07:59.604725 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.597327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:39101 (local address 127.0.0.1:39464)
0929 01:07:59.597560 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:59.597581 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:59.597719 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:59.598553 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:59.598559 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:59.598593 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:07:59.598789 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.598800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.600190 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.600196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.602248 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.602264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.603872 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.603877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.604091 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.604128 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 01:07:59.604247 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 01:07:59.604393 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 01:07:59.604751 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.597545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.5.198.252:39101)
0929 01:07:59.598234 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:59.598239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:59.598255 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:07:59.598311 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:59.598314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:59.598376 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:59.598472 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:07:59.599004 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.600030 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.602417 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.603662 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.603837 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.604114 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:59.604269 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 01:07:59.604392 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":440,"threads_started":1}
I20260929 01:07:59.605839 12853 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 01:07:59.606459 12881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03087ec95dd74fec8f8ccd4af29de874. Latest consensus state: current_term: 1 leader_uuid: "03087ec95dd74fec8f8ccd4af29de874" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true } } }
I20260929 01:07:59.606777 12881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [sys.catalog]: This master's current role is: LEARNER
W20260929 01:07:59.656715 12821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:07:59.656980 12821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:07:59.657521 12821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:07:59.664692 12821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:07:59.664811 12821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:07:59.664987 12821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:07:59.674480 12821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.193.keytab
--principal=oryx/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:35399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=37325
--tserver_master_addrs=127.5.198.254:39173,127.5.198.253:44255,127.5.198.252:39101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:07:59.675822 12821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:07:59.677189 12821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:07:59.690989 12899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:59.690999 12898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:07:59.690934 12901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:07:59.692049 12821 server_base.cc:1034] running on GCE node
I20260929 01:07:59.692726 12821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:07:59.693626 12821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:07:59.694855 12821 hybrid_clock.cc:648] HybridClock initialized: now 1790644079694803 us; error 69 us; skew 500 ppm
I20260929 01:07:59.695159 12821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:07:59 dist-test-slave-41cl krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644079, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:07:59.698808 12821 init.cc:377] Logged in from keytab as oryx/127.5.198.193@KRBTEST.COM (short username oryx)
I20260929 01:07:59.700182 12821 webserver.cc:460] Webserver started at http://127.5.198.193:37325/ using document root <none> and password file <none>
I20260929 01:07:59.700906 12821 fs_manager.cc:359] Metadata directory not provided
I20260929 01:07:59.701041 12821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:07:59.706112 12821 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:07:59.708809 12908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:59.710114 12821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:07:59.710283 12821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "4cd3434ac2e041c299afa3fcfd3031ac"
format_stamp: "Formatted at 2026-09-29 01:07:56 on dist-test-slave-41cl"
server_key: "c6bec09be8de2aa0d34c954885d67c5b"
server_key_iv: "9c3ff93b960cfc161c5888b968ad9b72"
server_key_version: "encryptionkey@0"
I20260929 01:07:59.711360 12821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:07:59.724005 12821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:07:59.728569 12821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:07:59.728826 12821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:07:59.729807 12821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:07:59.731105 12821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:07:59.731184 12821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:59.731325 12821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:07:59.731397 12821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:07:59.764780 12821 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:35399
I20260929 01:07:59.764853 13020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:35399 every 8 connection(s)
I20260929 01:07:59.766150 12821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:07:59.771041 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 12821
I20260929 01:07:59.771381 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 12252
Sep 29 01:07:59 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644079, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.252@KRBTEST.COM
Sep 29 01:07:59 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644079, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM
Sep 29 01:07:59 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644079, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.253@KRBTEST.COM
Sep 29 01:07:59 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.198.194@KRBTEST.COM
I20260929 01:07:59.794555 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.769686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:38727 (local address 127.5.198.252:39101)
0929 01:07:59.770089 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:59.770097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:59.771067 (+ 970us) server_negotiation.cc:407] Connection header received
0929 01:07:59.773933 (+ 2866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:59.773939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:59.774025 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:59.774096 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:59.776929 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.778483 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.780929 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.781556 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.781642 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.787354 (+ 5712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:59.787361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:59.787368 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:59.787408 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:59.790876 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.791766 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.791772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.791777 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.791951 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.792510 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.792515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.792519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.792876 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:59.793025 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:59.793915 (+ 890us) server_negotiation.cc:299] Negotiation successful
0929 01:07:59.794185 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
I20260929 01:07:59.795753 13025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.769563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:39101 (local address 127.5.198.193:38727)
0929 01:07:59.770914 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:59.770959 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:59.773605 (+ 2646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:59.774543 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:59.774554 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:59.775466 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:59.776708 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.776733 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.778682 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.778694 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.780753 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.780766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.782006 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.782015 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.782340 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.783555 (+ 1215us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:59.783579 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:59.787098 (+ 3519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:59.791036 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.791046 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.791065 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.791626 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.792099 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.792104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.792109 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.792352 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.792999 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:59.793012 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:59.793466 (+ 454us) client_negotiation.cc:769] Sending connection context
0929 01:07:59.793776 (+ 310us) client_negotiation.cc:240] Negotiation successful
0929 01:07:59.794296 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"mutex_wait_us":315,"spinlock_wait_cycles":68608,"thread_start_us":3620,"threads_started":1}
I20260929 01:07:59.796437 13029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.771068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:45553 (local address 127.5.198.253:44255)
0929 01:07:59.774710 (+ 3642us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:59.774715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:59.774739 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:07:59.774802 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:59.774807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:59.774880 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:59.774987 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:59.776942 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.778140 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.780954 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.781585 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.781705 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.788877 (+ 7172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:59.788885 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:59.788889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:59.788931 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:59.792256 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.793201 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.793205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.793208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.793345 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.793791 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.793795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.793798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.794107 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:59.794242 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:59.795840 (+ 1598us) server_negotiation.cc:299] Negotiation successful
0929 01:07:59.796048 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3539,"thread_start_us":3427,"threads_started":1}
I20260929 01:07:59.796767 13028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.770716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:44255 (local address 127.5.198.193:45553)
0929 01:07:59.774127 (+ 3411us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:59.774150 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:59.774288 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:59.775207 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:59.775213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:59.775864 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:59.776709 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.776735 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.778310 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.778316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.780795 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.780808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.782009 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.782017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.782420 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.783377 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:59.783414 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:59.788667 (+ 5253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:59.792531 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.792534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.792537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.793073 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.793493 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.793500 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.793505 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.793666 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.794296 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:59.794301 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:59.794428 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 01:07:59.794565 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 01:07:59.794805 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3267,"mutex_wait_us":467,"spinlock_wait_cycles":34048,"thread_start_us":3119,"threads_started":1}
I20260929 01:07:59.797446 13021 heartbeater.cc:347] Connected to a master server at 127.5.198.252:39101
I20260929 01:07:59.797787 13021 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:59.798067 13023 heartbeater.cc:347] Connected to a master server at 127.5.198.253:44255
I20260929 01:07:59.798197 13023 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:59.799014 13023 heartbeater.cc:511] Master 127.5.198.253:44255 requested a full tablet report, sending...
I20260929 01:07:59.800160 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.769564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.193:37749)
0929 01:07:59.772096 (+ 2532us) negotiation.cc:107] Waiting for socket to connect
0929 01:07:59.772118 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:07:59.773605 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:07:59.774543 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:07:59.774563 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:07:59.775462 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:07:59.776910 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.776921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.778286 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.778291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.780795 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:07:59.780808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.782027 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.782036 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:07:59.782402 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.783389 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:07:59.783415 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:07:59.787257 (+ 3842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:07:59.792413 (+ 5156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.792419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.792422 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.796498 (+ 4076us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.798356 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:07:59.798361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:07:59.798364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:07:59.798536 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:07:59.799184 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:07:59.799189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:07:59.799315 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 01:07:59.799488 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 01:07:59.799747 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2379,"mutex_wait_us":541,"spinlock_wait_cycles":8448,"thread_start_us":789,"threads_started":1}
I20260929 01:07:59.800182 13026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.769835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:37749 (local address 127.5.198.254:39173)
0929 01:07:59.771589 (+ 1754us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:59.771594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:59.772200 (+ 606us) server_negotiation.cc:407] Connection header received
0929 01:07:59.774032 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:59.774037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:59.774114 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:59.774205 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:59.777090 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.778162 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.780951 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.781589 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.781682 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.787576 (+ 5894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:59.787582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:59.787586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:59.787625 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:59.791101 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.798011 (+ 6910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.798019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.798024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.798222 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.798654 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.798659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.798663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.798989 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:59.799153 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:59.799595 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 01:07:59.799834 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1633,"thread_start_us":1539,"threads_started":1}
I20260929 01:07:59.801131 13031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:07:59.772914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44320 (local address 127.5.198.194:42701)
0929 01:07:59.773509 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0929 01:07:59.773518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:07:59.773543 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:07:59.774655 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:07:59.774667 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:07:59.774890 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:07:59.777091 (+ 2201us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:07:59.777327 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.783325 (+ 5998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.785293 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:07:59.786014 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:07:59.786133 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:07:59.793283 (+ 7150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:07:59.793293 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:07:59.793305 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:07:59.793337 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:07:59.796935 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.798256 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.798265 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.798268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.798417 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:07:59.798820 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:07:59.798827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:07:59.798830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:07:59.799246 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:07:59.799530 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 01:07:59.800645 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0929 01:07:59.800860 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":572,"threads_started":1}
I20260929 01:07:59.801503 12655 ts_manager.cc:194] Registered new tserver with Master: 4cd3434ac2e041c299afa3fcfd3031ac (127.5.198.193:35399)
I20260929 01:07:59.800621 13021 heartbeater.cc:511] Master 127.5.198.252:39101 requested a full tablet report, sending...
I20260929 01:07:59.802492 12375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:07:59.803215 13022 heartbeater.cc:347] Connected to a master server at 127.5.198.254:39173
I20260929 01:07:59.803377 13022 heartbeater.cc:464] Registering TS with master...
I20260929 01:07:59.803877 13022 heartbeater.cc:511] Master 127.5.198.254:39173 requested a full tablet report, sending...
I20260929 01:07:59.804294 12843 ts_manager.cc:194] Registered new tserver with Master: 4cd3434ac2e041c299afa3fcfd3031ac (127.5.198.193:35399)
I20260929 01:07:59.805156 12057 ts_manager.cc:194] Re-registered known tserver with Master: 4cd3434ac2e041c299afa3fcfd3031ac (127.5.198.193:35399)
I20260929 01:07:59.806216 12057 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:37749
I20260929 01:07:59.890655 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 12252
I20260929 01:07:59.907465 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:42701
--local_ip_for_outbound_sockets=127.5.198.194
--tserver_master_addrs=127.5.198.254:39173
--webserver_port=36431
--webserver_interface=127.5.198.194
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.194.keytab
--principal=oryx/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.198.254:39173,127.5.198.253:44255
--tserver_master_addrs=127.5.198.254:39173,127.5.198.253:44255,127.5.198.252:39101 with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:08:00.218849 13034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03087ec95dd74fec8f8ccd4af29de874" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: NON_VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: true } } }
I20260929 01:08:00.219199 13034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:00.220218 13034 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874: attempting to promote NON_VOTER 9ecc84047a304e718e1f8e6eb02f0a74 to VOTER
I20260929 01:08:00.220825 13034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } }
I20260929 01:08:00.222823 12664 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 1 FOLLOWER]: Refusing update from remote peer 03087ec95dd74fec8f8ccd4af29de874: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 01:08:00.222810 12853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 1 LEARNER]: Refusing update from remote peer 03087ec95dd74fec8f8ccd4af29de874: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 01:08:00.223404 13034 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 01:08:00.223776 13036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 01:08:00.228101 13034 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9ecc84047a304e718e1f8e6eb02f0a74 (127.5.198.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } } }
I20260929 01:08:00.232038 13034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "03087ec95dd74fec8f8ccd4af29de874" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } } }
I20260929 01:08:00.232273 13034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:00.233180 13034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03087ec95dd74fec8f8ccd4af29de874" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } } }
I20260929 01:08:00.233395 13034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:00.232823 12853 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9ecc84047a304e718e1f8e6eb02f0a74 (127.5.198.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } } }
I20260929 01:08:00.233909 13034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03087ec95dd74fec8f8ccd4af29de874" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } } }
I20260929 01:08:00.234133 13034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874 [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:00.234503 12664 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9ecc84047a304e718e1f8e6eb02f0a74 (127.5.198.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } } }
I20260929 01:08:00.237810 13044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03087ec95dd74fec8f8ccd4af29de874" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } } }
I20260929 01:08:00.238220 13044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:08:00.238636 13045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03087ec95dd74fec8f8ccd4af29de874" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } } }
I20260929 01:08:00.239024 13045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:08:00.244057 13033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:00.244333 13033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:00.244428 13033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:00.251955 13033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:00.252058 13033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:00.252169 13033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:08:00.262272 13033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.194.keytab
--principal=oryx/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:42701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=36431
--tserver_master_addrs=127.5.198.254:39173,127.5.198.253:44255,127.5.198.252:39101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:00.263509 13033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:00.264748 13033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:00.277714 13048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:00.277776 13050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:00.277814 13047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:00.279443 13033 server_base.cc:1034] running on GCE node
I20260929 01:08:00.280056 13033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:00.281055 13033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:00.282285 13033 hybrid_clock.cc:648] HybridClock initialized: now 1790644080282241 us; error 52 us; skew 500 ppm
I20260929 01:08:00.282598 13033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:00 dist-test-slave-41cl krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644080, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:00.285792 13033 init.cc:377] Logged in from keytab as oryx/127.5.198.194@KRBTEST.COM (short username oryx)
I20260929 01:08:00.287278 13033 webserver.cc:460] Webserver started at http://127.5.198.194:36431/ using document root <none> and password file <none>
I20260929 01:08:00.287966 13033 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:00.288098 13033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:00.293265 13033 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:08:00.295887 13057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:00.297328 13033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:08:00.297550 13033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "618b7791d9e34150811fd7af3fb00c5a"
format_stamp: "Formatted at 2026-09-29 01:07:57 on dist-test-slave-41cl"
server_key: "5bf5cae55abf5b315c629fadefaab8f8"
server_key_iv: "0d3d829c55aa381d194880f1d3d0cc4f"
server_key_version: "encryptionkey@0"
I20260929 01:08:00.298527 13033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:00.314002 13033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:00.318358 13033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:00.318631 13033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:00.319519 13033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:00.320912 13033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:00.321026 13033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:00.321158 13033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:00.321246 13033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:00.354547 13033 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:42701
I20260929 01:08:00.354601 13169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:42701 every 8 connection(s)
I20260929 01:08:00.355953 13033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:08:00.359143 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 13033
I20260929 01:08:00.359475 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 12385
Sep 29 01:08:00 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644080, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.253@KRBTEST.COM
Sep 29 01:08:00 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.198.195@KRBTEST.COM
I20260929 01:08:00.395330 13174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:00.358906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:43923 (local address 127.5.198.253:44255)
0929 01:08:00.369658 (+ 10752us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:00.369668 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:00.373433 (+ 3765us) server_negotiation.cc:407] Connection header received
0929 01:08:00.375791 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:00.375799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:00.375898 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:00.375982 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:00.378332 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.379550 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.381932 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.382526 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.382621 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:00.387487 (+ 4866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:00.387493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:00.387498 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:00.387544 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:00.391654 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:00.392660 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:00.392667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:00.392673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:00.392804 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:00.393315 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:00.393320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:00.393323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:00.393686 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:00.393827 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:00.394738 (+ 911us) server_negotiation.cc:299] Negotiation successful
0929 01:08:00.394959 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10585,"thread_start_us":16885,"threads_started":1}
I20260929 01:08:00.397468 13180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:00.371889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:44255 (local address 127.5.198.194:43923)
0929 01:08:00.373267 (+ 1378us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:00.373308 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:00.375465 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:00.376422 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:00.376432 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:00.377290 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:00.378122 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:00.378148 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.379736 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.379741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:00.381773 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:00.381787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.382938 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.382945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:00.383258 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:00.384036 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:00.384078 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:00.387217 (+ 3139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:00.391828 (+ 4611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:00.391838 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:00.391852 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:00.392526 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:00.392959 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:00.392965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:00.392968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:00.393189 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:00.393915 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:00.393924 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:00.394320 (+ 396us) client_negotiation.cc:769] Sending connection context
0929 01:08:00.395619 (+ 1299us) client_negotiation.cc:240] Negotiation successful
0929 01:08:00.396096 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"threads_started":1}
I20260929 01:08:00.398284 13177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:00.361095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54596 (local address 127.5.198.195:35401)
0929 01:08:00.370192 (+ 9097us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:00.370204 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:00.375944 (+ 5740us) server_negotiation.cc:407] Connection header received
0929 01:08:00.376259 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:00.376268 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:00.376532 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:00.376694 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:00.378927 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.380392 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.382164 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.382981 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.383101 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:00.390331 (+ 7230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:00.390345 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:00.390363 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:00.390402 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:00.394068 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:00.394856 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:00.394865 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:00.394870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:00.395028 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:00.395665 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:00.395670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:00.395673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:00.396040 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:00.396284 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:00.397419 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0929 01:08:00.398017 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8898,"thread_start_us":8706,"threads_started":1}
I20260929 01:08:00.399556 12508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:00.400743 13172 heartbeater.cc:347] Connected to a master server at 127.5.198.253:44255
I20260929 01:08:00.401064 13172 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:00.402154 13172 heartbeater.cc:511] Master 127.5.198.253:44255 requested a full tablet report, sending...
I20260929 01:08:00.404662 12655 ts_manager.cc:194] Registered new tserver with Master: 618b7791d9e34150811fd7af3fb00c5a (127.5.198.194:42701)
Sep 29 01:08:00 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644080, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM
Sep 29 01:08:00 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644080, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.252@KRBTEST.COM
I20260929 01:08:00.417325 13176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:00.359417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:47591 (local address 127.5.198.254:39173)
0929 01:08:00.399711 (+ 40294us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:00.399717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:00.399737 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:08:00.399802 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:00.399810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:00.399921 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:00.401502 (+ 1581us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:00.401707 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.403106 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.405872 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.406593 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.406716 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:00.409806 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:00.409814 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:00.409823 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:00.409877 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:00.413817 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:00.414871 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:00.414876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:00.414884 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:00.415001 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:00.415781 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:00.415785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:00.415787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:00.416107 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:00.416260 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:00.416656 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 01:08:00.416948 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40151,"thread_start_us":40051,"threads_started":1}
I20260929 01:08:00.417553 13181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:00.372598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.194:47591)
0929 01:08:00.374555 (+ 1957us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:00.374572 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:00.375465 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:00.400147 (+ 24682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:00.400154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:00.400972 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:00.401310 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:00.401325 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.404348 (+ 3023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.404354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:00.405693 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:00.405704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.406769 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.406775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:00.406956 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:00.407550 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:00.407567 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:00.409616 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:00.414057 (+ 4441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:00.414062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:00.414067 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:00.414662 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:00.415132 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:00.415139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:00.415145 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:00.415293 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:00.416230 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:00.416235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:00.416381 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 01:08:00.416549 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 01:08:00.417080 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1816,"thread_start_us":347,"threads_started":1}
I20260929 01:08:00.418963 13171 heartbeater.cc:347] Connected to a master server at 127.5.198.254:39173
I20260929 01:08:00.419112 13171 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:00.419473 13171 heartbeater.cc:511] Master 127.5.198.254:39173 requested a full tablet report, sending...
I20260929 01:08:00.420380 12057 ts_manager.cc:194] Re-registered known tserver with Master: 618b7791d9e34150811fd7af3fb00c5a (127.5.198.194:42701)
I20260929 01:08:00.421444 12057 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:47591
I20260929 01:08:00.423250 13180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:00.371890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:39101 (local address 127.5.198.194:58583)
0929 01:08:00.397764 (+ 25874us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:00.397782 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:00.397869 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:00.405449 (+ 7580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:00.405454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:00.405921 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:00.406125 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:00.406135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.408579 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.408585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:00.410704 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:00.410718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.412049 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.412056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:00.412291 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:00.412878 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:00.412899 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:00.415660 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:00.420605 (+ 4945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:00.420609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:00.420612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:00.421053 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:00.421494 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:00.421499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:00.421502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:00.421709 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:00.422305 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:00.422310 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:00.422466 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 01:08:00.422595 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 01:08:00.422863 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25737,"mutex_wait_us":1923,"thread_start_us":379,"threads_started":1}
I20260929 01:08:00.423271 13175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:00.359067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:58583 (local address 127.5.198.252:39101)
0929 01:08:00.403699 (+ 44632us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:00.403709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:00.403732 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:08:00.403801 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:00.403809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:00.403913 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:00.404085 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:00.407126 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.408370 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.410864 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.411629 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.411804 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:00.415888 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:00.415896 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:00.415901 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:00.415951 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:00.420448 (+ 4497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:00.421186 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:00.421193 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:00.421197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:00.421360 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:00.421844 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:00.421850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:00.421854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:00.422175 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:00.422349 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:00.422709 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 01:08:00.422900 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44461,"thread_start_us":44277,"threads_started":1}
I20260929 01:08:00.424201 13170 heartbeater.cc:347] Connected to a master server at 127.5.198.252:39101
I20260929 01:08:00.424346 13170 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:00.424717 13170 heartbeater.cc:511] Master 127.5.198.252:39101 requested a full tablet report, sending...
I20260929 01:08:00.425666 12843 ts_manager.cc:194] Registered new tserver with Master: 618b7791d9e34150811fd7af3fb00c5a (127.5.198.194:42701)
I20260929 01:08:00.493670 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 12385
I20260929 01:08:00.510382 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:35401
--local_ip_for_outbound_sockets=127.5.198.195
--tserver_master_addrs=127.5.198.254:39173
--webserver_port=34235
--webserver_interface=127.5.198.195
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.195.keytab
--principal=oryx/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.198.254:39173,127.5.198.253:44255
--tserver_master_addrs=127.5.198.254:39173,127.5.198.253:44255,127.5.198.252:39101 with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:08:00.808959 13022 heartbeater.cc:503] Master 127.5.198.254:39173 was elected leader, sending a full tablet report...
W20260929 01:08:00.839735 13183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:00.839990 13183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:00.840080 13183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:00.847481 13183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:00.847579 13183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:00.847685 13183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:08:00.857648 13183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.195.keytab
--principal=oryx/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:35401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=34235
--tserver_master_addrs=127.5.198.254:39173,127.5.198.253:44255,127.5.198.252:39101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:00.859078 13183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:00.860297 13183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:00.873108 13188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:00.873517 13189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:00.873808 13191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:00.874254 13183 server_base.cc:1034] running on GCE node
I20260929 01:08:00.874790 13183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:00.875694 13183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:00.876883 13183 hybrid_clock.cc:648] HybridClock initialized: now 1790644080876840 us; error 38 us; skew 500 ppm
I20260929 01:08:00.877199 13183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:00 dist-test-slave-41cl krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644080, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:00.880298 13183 init.cc:377] Logged in from keytab as oryx/127.5.198.195@KRBTEST.COM (short username oryx)
I20260929 01:08:00.881662 13183 webserver.cc:460] Webserver started at http://127.5.198.195:34235/ using document root <none> and password file <none>
I20260929 01:08:00.882315 13183 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:00.882478 13183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:00.887566 13183 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260929 01:08:00.890158 13198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:00.891441 13183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:08:00.891626 13183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "59b3bc938cb34499a7aa7607b65ea443"
format_stamp: "Formatted at 2026-09-29 01:07:57 on dist-test-slave-41cl"
server_key: "f7794d243d99a652be18e79255ebde13"
server_key_iv: "d86918e034f793fb424b79c1e560bc11"
server_key_version: "encryptionkey@0"
I20260929 01:08:00.892587 13183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:00.914745 13183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:00.919318 13183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:00.919598 13183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:00.920579 13183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:00.921864 13183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:00.921957 13183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:00.922029 13183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:00.922134 13183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:00.952911 13183 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:35401
I20260929 01:08:00.953027 13310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:35401 every 8 connection(s)
I20260929 01:08:00.954312 13183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:08:00.960299 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 13183
Sep 29 01:08:00 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644080, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM
Sep 29 01:08:00 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644080, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.252@KRBTEST.COM
Sep 29 01:08:00 dist-test-slave-41cl krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644080, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.253@KRBTEST.COM
I20260929 01:08:00.990049 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:00.957142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:42771 (local address 127.5.198.254:39173)
0929 01:08:00.957630 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:00.957637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:00.962571 (+ 4934us) server_negotiation.cc:407] Connection header received
0929 01:08:00.963274 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:00.963279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:00.963372 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:00.963501 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:00.966554 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.968049 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.972372 (+ 4323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.973044 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.973174 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:00.978229 (+ 5055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:00.978239 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:00.978247 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:00.978306 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:00.983499 (+ 5193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:00.985914 (+ 2415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:00.985921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:00.985948 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:00.986139 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:00.987191 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:00.987199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:00.987204 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:00.987678 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:00.987844 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:00.989204 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0929 01:08:00.989502 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":211,"threads_started":1}
I20260929 01:08:00.991156 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:00.957708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.195:42771)
0929 01:08:00.959572 (+ 1864us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:00.959640 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:00.962953 (+ 3313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:00.963750 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:00.963766 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:00.964919 (+ 1153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:00.966325 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:00.966355 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.970746 (+ 4391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.970751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:00.972226 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:00.972239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.973230 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.973237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:00.973405 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:00.974011 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:00.974031 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:00.977944 (+ 3913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:00.983640 (+ 5696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:00.983653 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:00.983672 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:00.984274 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:00.986250 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:00.986256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:00.986263 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:00.986545 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:00.987822 (+ 1277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:00.987840 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:00.988296 (+ 456us) client_negotiation.cc:769] Sending connection context
0929 01:08:00.988645 (+ 349us) client_negotiation.cc:240] Negotiation successful
0929 01:08:00.989244 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1484,"mutex_wait_us":161,"spinlock_wait_cycles":14720,"thread_start_us":400,"threads_started":1}
I20260929 01:08:00.992874 13312 heartbeater.cc:347] Connected to a master server at 127.5.198.254:39173
I20260929 01:08:00.993201 13312 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:00.993727 13318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:00.958508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:52319 (local address 127.5.198.252:39101)
0929 01:08:00.958998 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:00.959006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:00.959889 (+ 883us) server_negotiation.cc:407] Connection header received
0929 01:08:00.963390 (+ 3501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:00.963398 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:00.963507 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:00.963612 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:00.966919 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.968476 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.970684 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.971307 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.971447 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:00.980212 (+ 8765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:00.980221 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:00.980228 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:00.980276 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:00.985709 (+ 5433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:00.987163 (+ 1454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:00.987170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:00.987176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:00.987322 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:00.991743 (+ 4421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:00.991748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:00.991752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:00.992162 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:00.992311 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:00.993135 (+ 824us) server_negotiation.cc:299] Negotiation successful
0929 01:08:00.993406 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":416,"threads_started":1}
I20260929 01:08:00.994205 13312 heartbeater.cc:511] Master 127.5.198.254:39173 requested a full tablet report, sending...
I20260929 01:08:00.994689 13320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:00.957708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:39101 (local address 127.5.198.195:52319)
0929 01:08:00.959781 (+ 2073us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:00.959799 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:00.962966 (+ 3167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:00.963859 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:00.963867 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:00.964907 (+ 1040us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:00.966713 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:00.966727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.968714 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.968722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:00.970536 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:00.970551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.971577 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.971586 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:00.971991 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:00.973859 (+ 1868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:00.973909 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:00.979969 (+ 6060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:00.986313 (+ 6344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:00.986321 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:00.986328 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:00.987012 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:00.987462 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:00.987468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:00.987474 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:00.987634 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:00.992368 (+ 4734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:00.992374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:00.992554 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 01:08:00.992701 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 01:08:00.994154 (+ 1453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1947,"mutex_wait_us":148,"spinlock_wait_cycles":129536,"thread_start_us":10784,"threads_started":1}
I20260929 01:08:00.996300 12057 ts_manager.cc:194] Re-registered known tserver with Master: 59b3bc938cb34499a7aa7607b65ea443 (127.5.198.195:35401)
I20260929 01:08:00.997439 12057 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:42771
I20260929 01:08:00.998311 13311 heartbeater.cc:347] Connected to a master server at 127.5.198.252:39101
I20260929 01:08:00.998427 13311 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:00.998901 13311 heartbeater.cc:511] Master 127.5.198.252:39101 requested a full tablet report, sending...
I20260929 01:08:00.999308 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:00.958813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:35047 (local address 127.5.198.253:44255)
0929 01:08:00.959317 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:00.959326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:00.969225 (+ 9899us) server_negotiation.cc:407] Connection header received
0929 01:08:00.969432 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:00.969457 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:00.969534 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:00.969641 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:00.971635 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.972709 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.975434 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.976356 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.976568 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:00.988001 (+ 11433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:00.988011 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:00.988019 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:00.988083 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:00.994082 (+ 5999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:00.996618 (+ 2536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:00.996624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:00.996627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:00.996801 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:00.997643 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:00.997648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:00.997651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:00.998018 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:00.998147 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:00.998732 (+ 585us) server_negotiation.cc:299] Negotiation successful
0929 01:08:00.998953 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":268,"threads_started":1}
I20260929 01:08:01.000180 12843 ts_manager.cc:194] Registered new tserver with Master: 59b3bc938cb34499a7aa7607b65ea443 (127.5.198.195:35401)
I20260929 01:08:01.003131 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:00.958189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:44255 (local address 127.5.198.195:35047)
0929 01:08:00.969073 (+ 10884us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:00.969106 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:00.969271 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:00.969698 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:00.969706 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:00.970436 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:00.970852 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:00.970868 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.972843 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.972851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:00.974325 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:00.974339 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.980595 (+ 6256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:00.980602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:00.980821 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:00.982409 (+ 1588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:00.982438 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:00.986106 (+ 3668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:00.994249 (+ 8143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:00.994255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:00.994261 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:00.996437 (+ 2176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:00.997333 (+ 896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:00.997338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:00.997342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:00.997514 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:00.998206 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:00.998211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:00.998371 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 01:08:00.998548 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 01:08:01.002720 (+ 4172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10713,"mutex_wait_us":3779,"thread_start_us":3289,"threads_started":1}
I20260929 01:08:01.003914 13313 heartbeater.cc:347] Connected to a master server at 127.5.198.253:44255
I20260929 01:08:01.004040 13313 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:01.004899 13313 heartbeater.cc:511] Master 127.5.198.253:44255 requested a full tablet report, sending...
I20260929 01:08:01.006372 12655 ts_manager.cc:194] Registered new tserver with Master: 59b3bc938cb34499a7aa7607b65ea443 (127.5.198.195:35401)
I20260929 01:08:01.014703 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:00.975412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.5.198.254:39173)
0929 01:08:00.990464 (+ 15052us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:00.990473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:00.991758 (+ 1285us) server_negotiation.cc:407] Connection header received
0929 01:08:00.991811 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:00.991818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:00.992131 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:00.992218 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:00.994496 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:00.995393 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.001463 (+ 6070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.002087 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.002208 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.007041 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:01.007049 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:01.007056 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:01.007097 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:01.010769 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:01.011981 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:01.011988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:01.011991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:01.012144 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:01.012688 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:01.012693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:01.012696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:01.013530 (+ 834us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:01.013793 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:01.014119 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 01:08:01.014345 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14893,"thread_start_us":26846,"threads_started":1}
I20260929 01:08:01.019420 13318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:00.997033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39472 (local address 127.5.198.252:39101)
0929 01:08:00.997214 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:00.997219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:01.003021 (+ 5802us) server_negotiation.cc:407] Connection header received
0929 01:08:01.003070 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:01.003077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:01.003152 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:01.003245 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:01.006008 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.007051 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.008455 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.008888 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.009033 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.011767 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:01.011771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:01.011774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:01.011801 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:01.015542 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:01.016268 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:01.016275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:01.016279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:01.016412 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:01.017337 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:01.017343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:01.017346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:01.017696 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:01.017946 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:01.018900 (+ 954us) server_negotiation.cc:299] Negotiation successful
0929 01:08:01.019102 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 01:08:01.022946 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:00.975742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.5.198.253:44255)
0929 01:08:00.999552 (+ 23810us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:00.999556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:01.002762 (+ 3206us) server_negotiation.cc:407] Connection header received
0929 01:08:01.002816 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:01.002822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:01.002922 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:01.003018 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:01.006636 (+ 3618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.007749 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.009899 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.010653 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.010771 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.016675 (+ 5904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:01.016680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:01.016684 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:01.016715 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:01.020165 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:01.020913 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:01.020919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:01.020923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:01.021069 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:01.021472 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:01.021478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:01.021482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:01.021755 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:01.022031 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:01.022381 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 01:08:01.022604 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23707,"thread_start_us":26377,"threads_started":1}
I20260929 01:08:01.019210 12057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:08:01.026777 12057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:08:01.063001 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:01.045050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:35401 (local address 127.0.0.1:54608)
0929 01:08:01.052955 (+ 7905us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:01.052972 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:01.053129 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:01.054309 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:01.054314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:01.054340 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:01.054775 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:01.054792 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.057488 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.057494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:01.059365 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:01.059376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.062211 (+ 2835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.062217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:01.062380 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.062413 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:08:01.062515 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 01:08:01.062637 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7781,"thread_start_us":9701,"threads_started":1}
I20260929 01:08:01.063058 13340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:01.045494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.5.198.195:35401)
0929 01:08:01.053650 (+ 8156us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:01.053658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:01.053697 (+ 39us) server_negotiation.cc:407] Connection header received
0929 01:08:01.053826 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:01.053831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:01.054110 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:01.054337 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:01.055294 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.056910 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.059525 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.061965 (+ 2440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.062097 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.062334 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:01.062584 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 01:08:01.062742 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7970,"thread_start_us":7803,"threads_started":1}
I20260929 01:08:01.065301 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:01.046109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.5.198.194:42701)
0929 01:08:01.047026 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:01.047038 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:01.055154 (+ 8116us) server_negotiation.cc:407] Connection header received
0929 01:08:01.055276 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:01.055286 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:01.055561 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:01.055800 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:01.056832 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.058428 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.062317 (+ 3889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.063882 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.064009 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.064191 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:01.064814 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 01:08:01.064979 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":652,"threads_started":1}
I20260929 01:08:01.066224 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:01.045528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:42701 (local address 127.0.0.1:44332)
0929 01:08:01.055011 (+ 9483us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:01.055035 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:01.055144 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:01.055787 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:01.055792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:01.055816 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:01.056614 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:01.056628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.059628 (+ 3000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.059633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:01.062150 (+ 2517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:01.062164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.064392 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.064398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:01.064594 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.064634 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 01:08:01.065716 (+ 1082us) client_negotiation.cc:240] Negotiation successful
0929 01:08:01.065857 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9368,"thread_start_us":7169,"threads_started":1}
I20260929 01:08:01.066417 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:01.056400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.5.198.193:35399)
0929 01:08:01.057399 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:01.057408 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:01.057922 (+ 514us) server_negotiation.cc:407] Connection header received
0929 01:08:01.058155 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:01.058161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:01.058311 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:01.058508 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:01.059119 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.060024 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.064066 (+ 4042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.065206 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.065360 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.065562 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:01.065983 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 01:08:01.066140 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":719,"threads_started":1}
I20260929 01:08:01.066259 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:01.055208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:35399 (local address 127.0.0.1:52182)
0929 01:08:01.057799 (+ 2591us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:01.057813 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:01.057997 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:01.058495 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:01.058500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:01.058522 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:01.058964 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:01.058972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.060149 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.060157 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:01.063158 (+ 3001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:01.063173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.065408 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.065414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:01.065594 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.065626 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 01:08:01.065709 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 01:08:01.065835 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2466,"thread_start_us":2261,"threads_started":1}
I20260929 01:08:01.067195 13246 tablet_service.cc:1467] Processing CreateTablet for tablet 6115d8118fc442698fe52a9df1514674 (DEFAULT_TABLE table=test-table [id=2e9ea3bc7a884e4a8aebef67d2588180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:01.068804 13246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6115d8118fc442698fe52a9df1514674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:01.070284 12956 tablet_service.cc:1467] Processing CreateTablet for tablet 6115d8118fc442698fe52a9df1514674 (DEFAULT_TABLE table=test-table [id=2e9ea3bc7a884e4a8aebef67d2588180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:01.070622 13105 tablet_service.cc:1467] Processing CreateTablet for tablet 6115d8118fc442698fe52a9df1514674 (DEFAULT_TABLE table=test-table [id=2e9ea3bc7a884e4a8aebef67d2588180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:01.071623 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6115d8118fc442698fe52a9df1514674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:01.071740 13105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6115d8118fc442698fe52a9df1514674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:01.083252 13344 tablet_bootstrap.cc:492] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a: Bootstrap starting.
I20260929 01:08:01.085249 13345 tablet_bootstrap.cc:492] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443: Bootstrap starting.
I20260929 01:08:01.087838 13344 tablet_bootstrap.cc:654] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:01.088516 13345 tablet_bootstrap.cc:654] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:01.089174 13346 tablet_bootstrap.cc:492] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac: Bootstrap starting.
I20260929 01:08:01.089531 13344 log.cc:824] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:08:01.090018 13345 log.cc:824] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443: Log is configured to *not* fsync() on all Append() calls
I20260929 01:08:01.092619 13344 tablet_bootstrap.cc:492] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a: No bootstrap required, opened a new log
I20260929 01:08:01.092871 13344 ts_tablet_manager.cc:1397] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260929 01:08:01.092869 13346 tablet_bootstrap.cc:654] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:01.093233 13345 tablet_bootstrap.cc:492] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443: No bootstrap required, opened a new log
I20260929 01:08:01.093489 13345 ts_tablet_manager.cc:1397] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 01:08:01.094031 13346 log.cc:824] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac: Log is configured to *not* fsync() on all Append() calls
I20260929 01:08:01.097064 13346 tablet_bootstrap.cc:492] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac: No bootstrap required, opened a new log
I20260929 01:08:01.097326 13346 ts_tablet_manager.cc:1397] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 01:08:01.101223 13344 raft_consensus.cc:348] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:01.101565 13344 raft_consensus.cc:374] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:01.101663 13344 raft_consensus.cc:729] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 618b7791d9e34150811fd7af3fb00c5a, State: Initialized, Role: FOLLOWER
I20260929 01:08:01.102284 13344 consensus_queue.cc:260] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:01.104231 13345 raft_consensus.cc:348] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:01.104521 13344 ts_tablet_manager.cc:1428] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260929 01:08:01.104583 13171 heartbeater.cc:503] Master 127.5.198.254:39173 was elected leader, sending a full tablet report...
I20260929 01:08:01.104669 13345 raft_consensus.cc:374] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:01.104787 13345 raft_consensus.cc:729] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b3bc938cb34499a7aa7607b65ea443, State: Initialized, Role: FOLLOWER
I20260929 01:08:01.105538 13345 consensus_queue.cc:260] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:01.108738 13345 ts_tablet_manager.cc:1428] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260929 01:08:01.110029 13312 heartbeater.cc:503] Master 127.5.198.254:39173 was elected leader, sending a full tablet report...
I20260929 01:08:01.109031 13346 raft_consensus.cc:348] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:01.111595 13346 raft_consensus.cc:374] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:01.112737 13346 raft_consensus.cc:729] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd3434ac2e041c299afa3fcfd3031ac, State: Initialized, Role: FOLLOWER
I20260929 01:08:01.113390 13346 consensus_queue.cc:260] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:01.115382 13346 ts_tablet_manager.cc:1428] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac: Time spent starting tablet: real 0.018s user 0.006s sys 0.008s
W20260929 01:08:01.123309 13173 tablet.cc:2369] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:08:01.206630 13314 tablet.cc:2369] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:08:01.270526 13024 tablet.cc:2369] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:08:01.320568 13352 raft_consensus.cc:482] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:08:01.320899 13352 raft_consensus.cc:504] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:01.322681 13352 leader_election.cc:288] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59b3bc938cb34499a7aa7607b65ea443 (127.5.198.195:35401), 618b7791d9e34150811fd7af3fb00c5a (127.5.198.194:42701)
I20260929 01:08:01.332296 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:01.323222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:51723 (local address 127.5.198.194:42701)
0929 01:08:01.323453 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:01.323461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:01.323984 (+ 523us) server_negotiation.cc:407] Connection header received
0929 01:08:01.324131 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:01.324135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:01.324212 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:01.324297 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:01.325108 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.326180 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.329577 (+ 3397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.330913 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.331105 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.331191 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:01.331927 (+ 736us) server_negotiation.cc:299] Negotiation successful
0929 01:08:01.332054 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:08:01.332289 13340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:01.323017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:40977 (local address 127.5.198.195:35401)
0929 01:08:01.323270 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:01.323279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:01.323803 (+ 524us) server_negotiation.cc:407] Connection header received
0929 01:08:01.324041 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:01.324045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:01.324136 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:01.324241 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:01.325108 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.326177 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.329577 (+ 3400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.330948 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.331105 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.331191 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:01.331862 (+ 671us) server_negotiation.cc:299] Negotiation successful
0929 01:08:01.332044 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:08:01.332563 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:01.322894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:35401 (local address 127.5.198.193:40977)
0929 01:08:01.323677 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:01.323702 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:01.323878 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:01.324292 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:01.324300 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:01.324328 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:01.324951 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:01.324966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.326319 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.326325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:01.329364 (+ 3039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:01.329379 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.331244 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.331249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:01.331589 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.331693 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 01:08:01.331803 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 01:08:01.331980 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":354,"thread_start_us":713,"threads_started":1}
I20260929 01:08:01.332598 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:01.322936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:42701 (local address 127.5.198.193:51723)
0929 01:08:01.323887 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:01.323902 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:01.324007 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:01.324319 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:01.324323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:01.324340 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:01.324955 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:01.324966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.326319 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.326325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:01.329364 (+ 3039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:01.329379 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.331227 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.331236 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:01.331594 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.331698 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 01:08:01.331881 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 01:08:01.332015 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":340,"thread_start_us":359,"threads_started":1}
I20260929 01:08:01.333554 13125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6115d8118fc442698fe52a9df1514674" candidate_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "618b7791d9e34150811fd7af3fb00c5a" is_pre_election: true
I20260929 01:08:01.333554 13266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6115d8118fc442698fe52a9df1514674" candidate_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b3bc938cb34499a7aa7607b65ea443" is_pre_election: true
I20260929 01:08:01.334031 13266 raft_consensus.cc:2454] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd3434ac2e041c299afa3fcfd3031ac in term 0.
I20260929 01:08:01.334057 13125 raft_consensus.cc:2454] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd3434ac2e041c299afa3fcfd3031ac in term 0.
I20260929 01:08:01.334653 12910 leader_election.cc:302] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd3434ac2e041c299afa3fcfd3031ac, 59b3bc938cb34499a7aa7607b65ea443; no voters:
I20260929 01:08:01.335053 13352 raft_consensus.cc:2781] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:08:01.335196 13352 raft_consensus.cc:482] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:08:01.335314 13352 raft_consensus.cc:3037] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:01.337620 13352 raft_consensus.cc:504] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:01.338258 13352 leader_election.cc:288] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [CANDIDATE]: Term 1 election: Requested vote from peers 59b3bc938cb34499a7aa7607b65ea443 (127.5.198.195:35401), 618b7791d9e34150811fd7af3fb00c5a (127.5.198.194:42701)
I20260929 01:08:01.338697 13266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6115d8118fc442698fe52a9df1514674" candidate_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b3bc938cb34499a7aa7607b65ea443"
I20260929 01:08:01.338897 13266 raft_consensus.cc:3037] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:01.338919 13125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6115d8118fc442698fe52a9df1514674" candidate_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "618b7791d9e34150811fd7af3fb00c5a"
I20260929 01:08:01.339128 13125 raft_consensus.cc:3037] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:01.341400 13266 raft_consensus.cc:2454] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd3434ac2e041c299afa3fcfd3031ac in term 1.
I20260929 01:08:01.341826 12910 leader_election.cc:302] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd3434ac2e041c299afa3fcfd3031ac, 59b3bc938cb34499a7aa7607b65ea443; no voters:
I20260929 01:08:01.342077 13125 raft_consensus.cc:2454] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd3434ac2e041c299afa3fcfd3031ac in term 1.
I20260929 01:08:01.342202 13352 raft_consensus.cc:2781] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:08:01.342772 13352 raft_consensus.cc:686] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [term 1 LEADER]: Becoming Leader. State: Replica: 4cd3434ac2e041c299afa3fcfd3031ac, State: Running, Role: LEADER
I20260929 01:08:01.343412 13352 consensus_queue.cc:237] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:01.349220 12056 catalog_manager.cc:5495] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd3434ac2e041c299afa3fcfd3031ac (127.5.198.193). New cstate: current_term: 1 leader_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } health_report { overall_health: HEALTHY } } }
I20260929 01:08:01.372993 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:01.367482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.5.198.193:35399)
0929 01:08:01.367645 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:01.367649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:01.367710 (+ 61us) server_negotiation.cc:407] Connection header received
0929 01:08:01.367970 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:01.367974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:01.368056 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:01.368217 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:08:01.368657 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.369584 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.370893 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.371279 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.371371 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.371766 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:08:01.372315 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:08:01.372421 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:01.372661 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 01:08:01.372723 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:08:01.382077 13125 raft_consensus.cc:1261] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a [term 1 FOLLOWER]: Refusing update from remote peer 4cd3434ac2e041c299afa3fcfd3031ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:08:01.382077 13266 raft_consensus.cc:1261] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443 [term 1 FOLLOWER]: Refusing update from remote peer 4cd3434ac2e041c299afa3fcfd3031ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:08:01.383116 13352 consensus_queue.cc:1035] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:08:01.383455 13355 consensus_queue.cc:1035] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:08:01.395329 13358 mvcc.cc:205] Tried to move back new op lower bound from 7334478157330030592 to 7334478157186760704. Current Snapshot: MvccSnapshot[applied={T|T < 7334478157330030592}]
I20260929 01:08:01.408247 13360 mvcc.cc:205] Tried to move back new op lower bound from 7334478157330030592 to 7334478157186760704. Current Snapshot: MvccSnapshot[applied={T|T < 7334478157330030592}]
I20260929 01:08:01.415194 12056 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:08:01.415462 12056 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:08:01.423647 12056 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03087ec95dd74fec8f8ccd4af29de874: Sending DeleteTablet for 3 replicas of tablet 6115d8118fc442698fe52a9df1514674
I20260929 01:08:01.424944 13105 tablet_service.cc:1514] Processing DeleteTablet for tablet 6115d8118fc442698fe52a9df1514674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:01 UTC) from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:44332
I20260929 01:08:01.424980 12956 tablet_service.cc:1514] Processing DeleteTablet for tablet 6115d8118fc442698fe52a9df1514674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:01 UTC) from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:52182
I20260929 01:08:01.425917 13246 tablet_service.cc:1514] Processing DeleteTablet for tablet 6115d8118fc442698fe52a9df1514674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:01 UTC) from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:54608
I20260929 01:08:01.426422 13245 tablet_service.cc:1514] Processing DeleteTablet for tablet 6115d8118fc442698fe52a9df1514674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:01 UTC) from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:54608
I20260929 01:08:01.427237 13368 tablet_replica.cc:330] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac: stopping tablet replica
I20260929 01:08:01.427697 13368 raft_consensus.cc:2229] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:08:01.428363 13368 raft_consensus.cc:2258] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:01.429172 13369 tablet_replica.cc:330] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a: stopping tablet replica
I20260929 01:08:01.429613 13369 raft_consensus.cc:2229] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:08:01.430122 13369 raft_consensus.cc:2258] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:01.432512 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 12025
I20260929 01:08:01.433902 12087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:01.440826 13368 ts_tablet_manager.cc:1905] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:01.445871 13369 ts_tablet_manager.cc:1905] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:01.451809 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:01.427199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47046 (local address 127.5.198.253:44255)
0929 01:08:01.427360 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:01.427366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:01.432070 (+ 4704us) server_negotiation.cc:407] Connection header received
0929 01:08:01.432367 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:01.432373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:01.432452 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:01.432555 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:01.435059 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.436318 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.438881 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.439543 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.439652 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.443599 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:01.443607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:01.443612 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:01.443655 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:01.448193 (+ 4538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:01.449211 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:01.449218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:01.449222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:01.449400 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:01.449853 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:01.449858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:01.449861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:01.450270 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:01.450503 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:01.451138 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 01:08:01.451395 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:08:01.454048 13369 ts_tablet_manager.cc:1918] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:08:01.454254 13370 tablet_replica.cc:330] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443: stopping tablet replica
I20260929 01:08:01.454290 13369 log.cc:1196] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/6115d8118fc442698fe52a9df1514674
I20260929 01:08:01.454854 13370 raft_consensus.cc:2229] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:08:01.454950 13369 ts_tablet_manager.cc:1939] T 6115d8118fc442698fe52a9df1514674 P 618b7791d9e34150811fd7af3fb00c5a: Deleting consensus metadata
I20260929 01:08:01.455523 13370 raft_consensus.cc:2258] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:01.456605 13368 ts_tablet_manager.cc:1918] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:08:01.456763 13368 log.cc:1196] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/6115d8118fc442698fe52a9df1514674
I20260929 01:08:01.457310 13368 ts_tablet_manager.cc:1939] T 6115d8118fc442698fe52a9df1514674 P 4cd3434ac2e041c299afa3fcfd3031ac: Deleting consensus metadata
I20260929 01:08:01.457969 13370 ts_tablet_manager.cc:1905] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:01.465706 13370 ts_tablet_manager.cc:1918] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:08:01.465857 13370 log.cc:1196] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/6115d8118fc442698fe52a9df1514674
I20260929 01:08:01.466401 13370 ts_tablet_manager.cc:1939] T 6115d8118fc442698fe52a9df1514674 P 59b3bc938cb34499a7aa7607b65ea443: Deleting consensus metadata
I20260929 01:08:01.471536 13318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:01.432024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39484 (local address 127.5.198.252:39101)
0929 01:08:01.433114 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:01.433123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:01.452904 (+ 19781us) server_negotiation.cc:407] Connection header received
0929 01:08:01.453148 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:01.453155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:01.453236 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:01.453308 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:01.455616 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.456859 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.459303 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.459995 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.460096 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.464056 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:01.464064 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:01.464069 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:01.464112 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:01.468041 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:01.469259 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:01.469265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:01.469270 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:01.469427 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:01.469905 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:01.469912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:01.469918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:01.470280 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:01.470557 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:01.470903 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 01:08:01.471135 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926}
I20260929 01:08:01.578107 12042 catalog_manager.cc:4841] TS 59b3bc938cb34499a7aa7607b65ea443 (127.5.198.195:35401): tablet 6115d8118fc442698fe52a9df1514674 (table test-table [id=2e9ea3bc7a884e4a8aebef67d2588180]) successfully deleted
I20260929 01:08:01.578852 12041 catalog_manager.cc:4841] TS 618b7791d9e34150811fd7af3fb00c5a (127.5.198.194:42701): tablet 6115d8118fc442698fe52a9df1514674 (table test-table [id=2e9ea3bc7a884e4a8aebef67d2588180]) successfully deleted
W20260929 01:08:01.579264 12042 catalog_manager.cc:4808] TS 59b3bc938cb34499a7aa7607b65ea443 (127.5.198.195:35401): delete failed for tablet 6115d8118fc442698fe52a9df1514674 because the tablet was not found. No further retry: Not found: Tablet not found: 6115d8118fc442698fe52a9df1514674
I20260929 01:08:01.579757 12042 catalog_manager.cc:4841] TS 4cd3434ac2e041c299afa3fcfd3031ac (127.5.198.193:35399): tablet 6115d8118fc442698fe52a9df1514674 (table test-table [id=2e9ea3bc7a884e4a8aebef67d2588180]) successfully deleted
I20260929 01:08:01.580200 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 12025
I20260929 01:08:01.637606 13318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:01.607709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39498 (local address 127.5.198.252:39101)
0929 01:08:01.610656 (+ 2947us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:01.610665 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:01.618211 (+ 7546us) server_negotiation.cc:407] Connection header received
0929 01:08:01.618763 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:01.618772 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:01.618879 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:01.618973 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:01.621723 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.623189 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.626534 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.627193 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.627322 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.630844 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:01.630850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:01.630855 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:01.630892 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:01.634372 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:01.635233 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:01.635240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:01.635244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:01.635389 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:01.635828 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:01.635833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:01.635837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:01.636188 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:01.636395 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:01.636997 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 01:08:01.637240 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2754}
I20260929 01:08:01.642421 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:01.607505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.5.198.253:44255)
0929 01:08:01.607968 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:01.607977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:01.626194 (+ 18217us) server_negotiation.cc:407] Connection header received
0929 01:08:01.626718 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:01.626724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:01.626804 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:01.626868 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:01.629127 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.630329 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.632182 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:01.632949 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:01.633044 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:01.636290 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:01.636295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:01.636301 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:01.636336 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:01.639949 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:01.640582 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:01.640589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:01.640592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:01.640740 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:01.641106 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:01.641111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:01.641114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:01.641370 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:01.641646 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:01.641921 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 01:08:01.642131 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260929 01:08:02.403302 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:02.402509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.194:46557)
0929 01:08:02.402990 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:02.403098 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":248,"threads_started":1}
W20260929 01:08:02.403680 13171 heartbeater.cc:644] Failed to heartbeat to 127.5.198.254:39173 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
I20260929 01:08:02.409935 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:02.409293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.193:45485)
0929 01:08:02.409677 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:02.409775 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":230,"threads_started":1}
W20260929 01:08:02.410461 13022 heartbeater.cc:644] Failed to heartbeat to 127.5.198.254:39173 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
I20260929 01:08:02.428566 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:02.427815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.195:38035)
0929 01:08:02.428203 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:02.428308 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":197,"threads_started":1}
W20260929 01:08:02.428944 13312 heartbeater.cc:644] Failed to heartbeat to 127.5.198.254:39173 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
I20260929 01:08:02.924386 13383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03087ec95dd74fec8f8ccd4af29de874)
I20260929 01:08:02.924682 13383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } }
I20260929 01:08:02.925839 13384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03087ec95dd74fec8f8ccd4af29de874)
I20260929 01:08:02.926000 13383 leader_election.cc:288] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03087ec95dd74fec8f8ccd4af29de874 (127.5.198.254:39173), 1bfef164acd2450381b461e655b0500a (127.5.198.253:44255)
I20260929 01:08:02.927027 13384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } }
I20260929 01:08:02.927480 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:02.926069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.0.0.1:48842)
0929 01:08:02.926790 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:02.927051 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":388,"threads_started":1}
W20260929 01:08:02.927717 12827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
I20260929 01:08:02.929812 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:02.929316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.0.0.1:48846)
0929 01:08:02.929547 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:02.929642 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260929 01:08:02.930035 12827 leader_election.cc:334] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 03087ec95dd74fec8f8ccd4af29de874 (127.5.198.254:39173): Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
I20260929 01:08:02.931169 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:02.928667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.0.0.1:48844)
0929 01:08:02.930656 (+ 1989us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:02.930870 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1871,"thread_start_us":1752,"threads_started":1}
I20260929 01:08:02.931102 13384 leader_election.cc:288] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03087ec95dd74fec8f8ccd4af29de874 (127.5.198.254:39173), 9ecc84047a304e718e1f8e6eb02f0a74 (127.5.198.252:39101)
W20260929 01:08:02.931392 12639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
I20260929 01:08:02.937474 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:02.926337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47058 (local address 127.5.198.253:44255)
0929 01:08:02.927047 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:02.927055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:02.927591 (+ 536us) server_negotiation.cc:407] Connection header received
0929 01:08:02.927866 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:02.927873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:02.927979 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:02.928090 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:02.928734 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:02.930152 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:02.933884 (+ 3732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:02.935725 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:02.935879 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:02.936036 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:02.936735 (+ 699us) server_negotiation.cc:299] Negotiation successful
0929 01:08:02.936955 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":373,"threads_started":1}
I20260929 01:08:02.938282 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:02.926211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:44255 (local address 127.0.0.1:47058)
0929 01:08:02.927498 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:02.927516 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:02.927693 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:02.928119 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:02.928125 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:02.928151 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:02.928514 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:02.928528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:02.930476 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:02.930484 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:02.933629 (+ 3145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:02.933651 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:02.936213 (+ 2562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:02.936221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:02.936458 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:02.936548 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 01:08:02.937732 (+ 1184us) client_negotiation.cc:240] Negotiation successful
0929 01:08:02.937897 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"thread_start_us":1047,"threads_started":1}
I20260929 01:08:02.938979 12664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1bfef164acd2450381b461e655b0500a" is_pre_election: true
I20260929 01:08:02.939360 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:02.938743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.0.0.1:48858)
0929 01:08:02.938918 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:02.939112 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 01:08:02.939358 12664 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ecc84047a304e718e1f8e6eb02f0a74 in term 1.
W20260929 01:08:02.939642 12639 leader_election.cc:334] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 03087ec95dd74fec8f8ccd4af29de874 (127.5.198.254:39173): Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
I20260929 01:08:02.940056 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:02.928831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:39101 (local address 127.0.0.1:39510)
0929 01:08:02.931369 (+ 2538us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:02.931392 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:02.931560 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:02.932285 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:02.932293 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:02.932327 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:02.932775 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:02.932788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:02.934760 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:02.934768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:02.937548 (+ 2780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:02.937563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:02.939305 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:02.939312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:02.939499 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:02.939542 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 01:08:02.939638 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 01:08:02.939795 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2455,"thread_start_us":8580,"threads_started":1}
I20260929 01:08:02.939936 12829 leader_election.cc:302] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bfef164acd2450381b461e655b0500a, 9ecc84047a304e718e1f8e6eb02f0a74; no voters: 03087ec95dd74fec8f8ccd4af29de874
I20260929 01:08:02.940230 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:02.929278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.5.198.252:39101)
0929 01:08:02.930997 (+ 1719us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:02.931002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:02.931463 (+ 461us) server_negotiation.cc:407] Connection header received
0929 01:08:02.931708 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:02.931716 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:02.931800 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:02.931891 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:02.932995 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:02.934552 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:02.937759 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:02.939029 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:02.939178 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:02.939281 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:02.939676 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 01:08:02.939805 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1593,"thread_start_us":1465,"threads_started":1}
I20260929 01:08:02.940374 13383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:08:02.940519 13383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 03087ec95dd74fec8f8ccd4af29de874)
I20260929 01:08:02.940613 13383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:08:02.941080 12853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1bfef164acd2450381b461e655b0500a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" is_pre_election: true
I20260929 01:08:02.942803 13383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } }
I20260929 01:08:02.943229 12853 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bfef164acd2450381b461e655b0500a in current term 2: Already voted for candidate 9ecc84047a304e718e1f8e6eb02f0a74 in this term.
I20260929 01:08:02.943497 13383 leader_election.cc:288] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [CANDIDATE]: Term 2 election: Requested vote from peers 03087ec95dd74fec8f8ccd4af29de874 (127.5.198.254:39173), 1bfef164acd2450381b461e655b0500a (127.5.198.253:44255)
I20260929 01:08:02.943995 12664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1bfef164acd2450381b461e655b0500a"
I20260929 01:08:02.944195 12664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:08:02.944257 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:02.943552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.0.0.1:48874)
0929 01:08:02.944031 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:02.944129 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376}
I20260929 01:08:02.944226 12640 leader_election.cc:302] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bfef164acd2450381b461e655b0500a; no voters: 03087ec95dd74fec8f8ccd4af29de874, 9ecc84047a304e718e1f8e6eb02f0a74
I20260929 01:08:02.945225 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:02.944845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.0.0.1:48878)
0929 01:08:02.944994 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:02.945077 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 01:08:02.945426 12827 leader_election.cc:334] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 03087ec95dd74fec8f8ccd4af29de874 (127.5.198.254:39173): Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
I20260929 01:08:02.946732 12664 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ecc84047a304e718e1f8e6eb02f0a74 in term 2.
I20260929 01:08:02.946897 13384 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:08:02.947126 12829 leader_election.cc:302] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bfef164acd2450381b461e655b0500a, 9ecc84047a304e718e1f8e6eb02f0a74; no voters: 03087ec95dd74fec8f8ccd4af29de874
I20260929 01:08:02.947400 13383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:08:02.947826 13383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [term 2 LEADER]: Becoming Leader. State: Replica: 9ecc84047a304e718e1f8e6eb02f0a74, State: Running, Role: LEADER
I20260929 01:08:02.948190 13383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } }
I20260929 01:08:02.949899 13393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ecc84047a304e718e1f8e6eb02f0a74. Latest consensus state: current_term: 2 leader_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } } }
I20260929 01:08:02.950176 13393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:02.950758 13395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:08:02.954761 13395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:08:02.955734 13395 catalog_manager.cc:1259] Loaded cluster ID: 55239c9c72aa4bf08cacc386bfc02c32
I20260929 01:08:02.955818 13395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:08:02.957232 13395 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:08:02.958005 13395 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: Loaded TSK: 0
I20260929 01:08:02.958863 13395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:08:02.981519 12842 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:08:02.988365 12842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:08:02.992100 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:02.991759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.0.0.1:48904)
0929 01:08:02.991898 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:02.991973 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 01:08:02.992280 12664 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [term 2 FOLLOWER]: Refusing update from remote peer 9ecc84047a304e718e1f8e6eb02f0a74: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 01:08:02.993103 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:02.992774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.0.0.1:48912)
0929 01:08:02.992891 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:02.992962 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 01:08:02.993127 13393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 01:08:02.993376 12827 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 -> Peer 03087ec95dd74fec8f8ccd4af29de874 (127.5.198.254:39173): Couldn't send request to peer 03087ec95dd74fec8f8ccd4af29de874. Status: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:08:02.995862 13384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ecc84047a304e718e1f8e6eb02f0a74. Latest consensus state: current_term: 2 leader_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } } }
I20260929 01:08:02.996096 13384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:08:02.997756 13383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } } }
I20260929 01:08:02.998147 13383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74 [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:02.998500 13397 mvcc.cc:205] Tried to move back new op lower bound from 7334478163930066944 to 7334478163759214592. Current Snapshot: MvccSnapshot[applied={T|T < 7334478163930066944}]
I20260929 01:08:02.998517 13384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "03087ec95dd74fec8f8ccd4af29de874" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39173 } } peers { permanent_uuid: "1bfef164acd2450381b461e655b0500a" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 44255 } attrs { promote: false } } peers { permanent_uuid: "9ecc84047a304e718e1f8e6eb02f0a74" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 39101 } attrs { promote: false } } }
I20260929 01:08:02.998778 13384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfef164acd2450381b461e655b0500a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:08:03.014470 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.006571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.5.198.195:35401)
0929 01:08:03.007853 (+ 1282us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:03.007862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:03.007885 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:08:03.007956 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:03.007961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:03.008029 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:03.008131 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:03.008715 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.009838 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.012193 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.013499 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.013622 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:03.013740 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:03.014062 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 01:08:03.014193 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1121,"thread_start_us":1004,"threads_started":1}
I20260929 01:08:03.014474 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.006501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:35401 (local address 127.0.0.1:54624)
0929 01:08:03.006689 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.006714 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:03.006820 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:03.008143 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:03.008152 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:03.008179 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:03.008526 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:03.008541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.009979 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.009984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:03.011975 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:03.011984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.013721 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.013728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:03.013889 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:03.013918 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 01:08:03.014040 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 01:08:03.014174 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260929 01:08:03.015879 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.008034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.5.198.193:35399)
0929 01:08:03.008849 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:03.008861 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:03.008878 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:08:03.008948 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:03.008952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:03.009018 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:03.009692 (+ 674us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:03.009849 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.010852 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.013414 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.014648 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.014794 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:03.014912 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:03.015430 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 01:08:03.015584 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":571,"threads_started":1}
I20260929 01:08:03.015955 13404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.007624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:35399 (local address 127.0.0.1:52204)
0929 01:08:03.008158 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.008178 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:03.008289 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:03.009178 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:03.009184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:03.009209 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:03.009550 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:03.009560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.010999 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.011007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:03.013266 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:03.013283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.015038 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.015044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:03.015224 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:03.015259 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 01:08:03.015344 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 01:08:03.015484 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":4,"thread_start_us":280,"threads_started":1}
I20260929 01:08:03.015964 13246 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf37b27e84b468f8f5d2004d8563c46 (DEFAULT_TABLE table=test-table [id=4f7b40ad11e1420583740ac8447fde97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:03.016407 13246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf37b27e84b468f8f5d2004d8563c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:03.016690 12956 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf37b27e84b468f8f5d2004d8563c46 (DEFAULT_TABLE table=test-table [id=4f7b40ad11e1420583740ac8447fde97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:03.017091 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf37b27e84b468f8f5d2004d8563c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:03.023087 13406 tablet_bootstrap.cc:492] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443: Bootstrap starting.
I20260929 01:08:03.023999 13407 tablet_bootstrap.cc:492] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac: Bootstrap starting.
I20260929 01:08:03.024920 13406 tablet_bootstrap.cc:654] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:03.025583 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.006823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:42701 (local address 127.0.0.1:44342)
0929 01:08:03.007016 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.007039 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:03.007160 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:03.011627 (+ 4467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:03.011632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:03.011654 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:03.011886 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:03.011902 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.013466 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.013471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:03.015327 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:03.015340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.024797 (+ 9457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.024803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:03.024998 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:03.025043 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 01:08:03.025131 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 01:08:03.025275 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":2}
I20260929 01:08:03.025599 13403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.007107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44342 (local address 127.5.198.194:42701)
0929 01:08:03.011312 (+ 4205us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:03.011318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:03.011335 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:08:03.011404 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:03.011408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:03.011478 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:03.012045 (+ 567us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:03.012245 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.013356 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.022738 (+ 9382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.024635 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.024781 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:03.024887 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:03.025156 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 01:08:03.025284 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4091,"thread_start_us":4007,"threads_started":1}
I20260929 01:08:03.025807 13407 tablet_bootstrap.cc:654] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:03.026494 13105 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf37b27e84b468f8f5d2004d8563c46 (DEFAULT_TABLE table=test-table [id=4f7b40ad11e1420583740ac8447fde97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:03.026873 13105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf37b27e84b468f8f5d2004d8563c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:03.027439 13406 tablet_bootstrap.cc:492] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443: No bootstrap required, opened a new log
I20260929 01:08:03.027611 13406 ts_tablet_manager.cc:1397] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 01:08:03.028640 13406 raft_consensus.cc:348] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:03.028985 13406 raft_consensus.cc:374] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:03.029101 13406 raft_consensus.cc:729] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b3bc938cb34499a7aa7607b65ea443, State: Initialized, Role: FOLLOWER
I20260929 01:08:03.029425 13406 consensus_queue.cc:260] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:03.030407 13406 ts_tablet_manager.cc:1428] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 01:08:03.033370 13311 heartbeater.cc:503] Master 127.5.198.252:39101 was elected leader, sending a full tablet report...
I20260929 01:08:03.038619 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.031197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.195:34697)
0929 01:08:03.038339 (+ 7142us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.038435 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7037,"thread_start_us":6939,"threads_started":1}
I20260929 01:08:03.038954 13412 tablet_bootstrap.cc:492] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a: Bootstrap starting.
I20260929 01:08:03.039690 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.039232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.195:47395)
0929 01:08:03.039386 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.039482 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47,"threads_started":1}
I20260929 01:08:03.039793 13407 tablet_bootstrap.cc:492] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac: No bootstrap required, opened a new log
I20260929 01:08:03.039947 13407 ts_tablet_manager.cc:1397] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
I20260929 01:08:03.040812 13407 raft_consensus.cc:348] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:03.041055 13407 raft_consensus.cc:374] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:03.041162 13407 raft_consensus.cc:729] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd3434ac2e041c299afa3fcfd3031ac, State: Initialized, Role: FOLLOWER
I20260929 01:08:03.041291 13412 tablet_bootstrap.cc:654] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:03.041409 13407 consensus_queue.cc:260] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:03.042271 13407 ts_tablet_manager.cc:1428] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:08:03.044191 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.042506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.193:60443)
0929 01:08:03.043824 (+ 1318us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.043973 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1188,"threads_started":1}
I20260929 01:08:03.044396 13412 tablet_bootstrap.cc:492] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a: No bootstrap required, opened a new log
I20260929 01:08:03.044430 13021 heartbeater.cc:503] Master 127.5.198.252:39101 was elected leader, sending a full tablet report...
I20260929 01:08:03.044593 13412 ts_tablet_manager.cc:1397] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260929 01:08:03.045387 13412 raft_consensus.cc:348] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:03.045612 13412 raft_consensus.cc:374] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:03.045720 13412 raft_consensus.cc:729] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 618b7791d9e34150811fd7af3fb00c5a, State: Initialized, Role: FOLLOWER
I20260929 01:08:03.046074 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.045029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.193:53349)
0929 01:08:03.045838 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.045930 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":619}
I20260929 01:08:03.045993 13412 consensus_queue.cc:260] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:03.046835 13412 ts_tablet_manager.cc:1428] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:08:03.048025 13418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.047079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.194:58907)
0929 01:08:03.047638 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.047786 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":273,"threads_started":1}
I20260929 01:08:03.048578 13170 heartbeater.cc:503] Master 127.5.198.252:39101 was elected leader, sending a full tablet report...
I20260929 01:08:03.049074 13418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.048764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.194:49175)
0929 01:08:03.048885 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.048952 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 01:08:03.241269 13414 raft_consensus.cc:482] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:08:03.241513 13414 raft_consensus.cc:504] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:03.242146 13414 leader_election.cc:288] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59b3bc938cb34499a7aa7607b65ea443 (127.5.198.195:35401), 618b7791d9e34150811fd7af3fb00c5a (127.5.198.194:42701)
I20260929 01:08:03.242767 13125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf37b27e84b468f8f5d2004d8563c46" candidate_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "618b7791d9e34150811fd7af3fb00c5a" is_pre_election: true
I20260929 01:08:03.242774 13266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf37b27e84b468f8f5d2004d8563c46" candidate_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b3bc938cb34499a7aa7607b65ea443" is_pre_election: true
I20260929 01:08:03.243021 13125 raft_consensus.cc:2454] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd3434ac2e041c299afa3fcfd3031ac in term 0.
I20260929 01:08:03.243037 13266 raft_consensus.cc:2454] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cd3434ac2e041c299afa3fcfd3031ac in term 0.
I20260929 01:08:03.243526 12909 leader_election.cc:302] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd3434ac2e041c299afa3fcfd3031ac, 618b7791d9e34150811fd7af3fb00c5a; no voters:
I20260929 01:08:03.243810 13414 raft_consensus.cc:2781] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:08:03.243925 13414 raft_consensus.cc:482] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:08:03.244069 13414 raft_consensus.cc:3037] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:03.246117 13414 raft_consensus.cc:504] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:03.246644 13414 leader_election.cc:288] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [CANDIDATE]: Term 1 election: Requested vote from peers 59b3bc938cb34499a7aa7607b65ea443 (127.5.198.195:35401), 618b7791d9e34150811fd7af3fb00c5a (127.5.198.194:42701)
I20260929 01:08:03.247125 13125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf37b27e84b468f8f5d2004d8563c46" candidate_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "618b7791d9e34150811fd7af3fb00c5a"
I20260929 01:08:03.247131 13266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf37b27e84b468f8f5d2004d8563c46" candidate_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b3bc938cb34499a7aa7607b65ea443"
I20260929 01:08:03.247290 13125 raft_consensus.cc:3037] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:03.247319 13266 raft_consensus.cc:3037] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:03.249488 13125 raft_consensus.cc:2454] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd3434ac2e041c299afa3fcfd3031ac in term 1.
I20260929 01:08:03.249496 13266 raft_consensus.cc:2454] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cd3434ac2e041c299afa3fcfd3031ac in term 1.
I20260929 01:08:03.249897 12910 leader_election.cc:302] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cd3434ac2e041c299afa3fcfd3031ac, 59b3bc938cb34499a7aa7607b65ea443; no voters:
I20260929 01:08:03.250193 13414 raft_consensus.cc:2781] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:08:03.250661 13414 raft_consensus.cc:686] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [term 1 LEADER]: Becoming Leader. State: Replica: 4cd3434ac2e041c299afa3fcfd3031ac, State: Running, Role: LEADER
I20260929 01:08:03.250993 13414 consensus_queue.cc:237] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } }
I20260929 01:08:03.251740 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.251242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.193:38335)
0929 01:08:03.251457 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.251547 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 01:08:03.253507 12843 catalog_manager.cc:5495] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cd3434ac2e041c299afa3fcfd3031ac (127.5.198.193). New cstate: current_term: 1 leader_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cd3434ac2e041c299afa3fcfd3031ac" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35399 } health_report { overall_health: HEALTHY } } }
I20260929 01:08:03.319235 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.314698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.5.198.193:35399)
0929 01:08:03.314895 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:03.314899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:03.314915 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:08:03.314994 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:03.314997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:03.315057 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:03.315128 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:08:03.315603 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.316370 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.317777 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.318212 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.318306 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:03.318675 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:08:03.318774 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:08:03.318839 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:03.318985 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 01:08:03.319043 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 01:08:03.322189 13125 raft_consensus.cc:1261] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a [term 1 FOLLOWER]: Refusing update from remote peer 4cd3434ac2e041c299afa3fcfd3031ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:08:03.322238 13266 raft_consensus.cc:1261] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443 [term 1 FOLLOWER]: Refusing update from remote peer 4cd3434ac2e041c299afa3fcfd3031ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:08:03.322839 13419 consensus_queue.cc:1035] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "618b7791d9e34150811fd7af3fb00c5a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:08:03.323176 13414 consensus_queue.cc:1035] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b3bc938cb34499a7aa7607b65ea443" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:08:03.325443 13418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.324943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.194:36197)
0929 01:08:03.325090 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.325218 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 01:08:03.326527 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.325688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.195:51359)
0929 01:08:03.325941 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.326124 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260929 01:08:03.330319 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.329873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.193:40709)
0929 01:08:03.330087 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.330179 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260929 01:08:03.333387 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.333012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.195:55967)
0929 01:08:03.333163 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.333242 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 01:08:03.337296 12843 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:08:03.337602 12843 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:08:03.346202 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.345770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.193:43153)
0929 01:08:03.345924 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.346034 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 01:08:03.347223 13418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.346769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.5.198.194:43715)
0929 01:08:03.346998 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.347088 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260929 01:08:03.348456 12843 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ecc84047a304e718e1f8e6eb02f0a74: Sending DeleteTablet for 3 replicas of tablet 6cf37b27e84b468f8f5d2004d8563c46
I20260929 01:08:03.349730 13105 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf37b27e84b468f8f5d2004d8563c46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:03 UTC) from {username='oryx', principal='oryx/127.5.198.252@KRBTEST.COM'} at 127.0.0.1:44342
I20260929 01:08:03.350791 13246 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf37b27e84b468f8f5d2004d8563c46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:03 UTC) from {username='oryx', principal='oryx/127.5.198.252@KRBTEST.COM'} at 127.0.0.1:54624
I20260929 01:08:03.351275 12956 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf37b27e84b468f8f5d2004d8563c46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:03 UTC) from {username='oryx', principal='oryx/127.5.198.252@KRBTEST.COM'} at 127.0.0.1:52204
I20260929 01:08:03.351438 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 12821
I20260929 01:08:03.352471 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf37b27e84b468f8f5d2004d8563c46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:03 UTC) from {username='oryx', principal='oryx/127.5.198.252@KRBTEST.COM'} at 127.0.0.1:52204
I20260929 01:08:03.368765 13433 tablet_replica.cc:330] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac: stopping tablet replica
I20260929 01:08:03.368968 13433 raft_consensus.cc:2229] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:08:03.369025 13432 tablet_replica.cc:330] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443: stopping tablet replica
I20260929 01:08:03.369204 13432 raft_consensus.cc:2229] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:08:03.368227 13431 tablet_replica.cc:330] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a: stopping tablet replica
I20260929 01:08:03.369513 13432 raft_consensus.cc:2258] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:03.369642 13431 raft_consensus.cc:2229] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:08:03.369982 13431 raft_consensus.cc:2258] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:03.370213 13433 raft_consensus.cc:2258] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:03.371117 13431 ts_tablet_manager.cc:1905] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:03.371232 13433 ts_tablet_manager.cc:1905] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:03.377664 13431 ts_tablet_manager.cc:1918] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:08:03.377852 13431 log.cc:1196] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/6cf37b27e84b468f8f5d2004d8563c46
I20260929 01:08:03.377899 13433 ts_tablet_manager.cc:1918] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:08:03.378046 13433 log.cc:1196] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/6cf37b27e84b468f8f5d2004d8563c46
I20260929 01:08:03.378445 13431 ts_tablet_manager.cc:1939] T 6cf37b27e84b468f8f5d2004d8563c46 P 618b7791d9e34150811fd7af3fb00c5a: Deleting consensus metadata
I20260929 01:08:03.378489 13433 ts_tablet_manager.cc:1939] T 6cf37b27e84b468f8f5d2004d8563c46 P 4cd3434ac2e041c299afa3fcfd3031ac: Deleting consensus metadata
I20260929 01:08:03.379516 12827 catalog_manager.cc:4841] TS 618b7791d9e34150811fd7af3fb00c5a (127.5.198.194:42701): tablet 6cf37b27e84b468f8f5d2004d8563c46 (table test-table [id=4f7b40ad11e1420583740ac8447fde97]) successfully deleted
I20260929 01:08:03.379676 12828 catalog_manager.cc:4841] TS 4cd3434ac2e041c299afa3fcfd3031ac (127.5.198.193:35399): tablet 6cf37b27e84b468f8f5d2004d8563c46 (table test-table [id=4f7b40ad11e1420583740ac8447fde97]) successfully deleted
W20260929 01:08:03.380688 12828 catalog_manager.cc:4808] TS 4cd3434ac2e041c299afa3fcfd3031ac (127.5.198.193:35399): delete failed for tablet 6cf37b27e84b468f8f5d2004d8563c46 because the tablet was not found. No further retry: Not found: Tablet not found: 6cf37b27e84b468f8f5d2004d8563c46
I20260929 01:08:03.381678 13432 ts_tablet_manager.cc:1905] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:03.384652 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.352505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.5.198.193:35399)
0929 01:08:03.352722 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:03.352730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:03.368529 (+ 15799us) server_negotiation.cc:407] Connection header received
0929 01:08:03.368744 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:03.368749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:03.368840 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:03.368949 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:03.371017 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.372120 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.373918 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.374586 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.374678 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:03.378395 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:03.378445 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:03.378472 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:03.378516 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:03.381944 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:03.382609 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:03.382619 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:03.382622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:03.382753 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:03.383144 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:03.383149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:03.383153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:03.383499 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:03.383790 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:03.384078 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 01:08:03.384276 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 01:08:03.385322 13016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:03.387845 13432 ts_tablet_manager.cc:1918] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:08:03.387975 13432 log.cc:1196] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/6cf37b27e84b468f8f5d2004d8563c46
I20260929 01:08:03.388368 13432 ts_tablet_manager.cc:1939] T 6cf37b27e84b468f8f5d2004d8563c46 P 59b3bc938cb34499a7aa7607b65ea443: Deleting consensus metadata
I20260929 01:08:03.389432 12828 catalog_manager.cc:4841] TS 59b3bc938cb34499a7aa7607b65ea443 (127.5.198.195:35401): tablet 6cf37b27e84b468f8f5d2004d8563c46 (table test-table [id=4f7b40ad11e1420583740ac8447fde97]) successfully deleted
I20260929 01:08:03.517660 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.517245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.0.0.1:59972)
0929 01:08:03.517413 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.517514 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 01:08:03.518955 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.518622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39173 (local address 127.0.0.1:59978)
0929 01:08:03.518765 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:03.518836 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:39173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 01:08:03.523065 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 12821
I20260929 01:08:03.545331 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 13033
I20260929 01:08:03.558851 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.546138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.5.198.194:42701)
0929 01:08:03.546620 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:03.546625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:03.546639 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:08:03.546689 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:03.546692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:03.546752 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:03.546867 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:03.548115 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.548910 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.550111 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.550481 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.550611 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:03.553122 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:03.553136 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:03.553152 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:03.553182 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:03.555943 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:03.556672 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:03.556679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:03.556682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:03.556806 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:03.557192 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:03.557199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:03.557203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:03.557599 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:03.557934 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:03.558214 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 01:08:03.558513 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":217,"threads_started":1}
I20260929 01:08:03.559583 13165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:03.651731 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 13033
I20260929 01:08:03.669595 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 13183
I20260929 01:08:03.682716 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:03.670342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37968 (local address 127.5.198.195:35401)
0929 01:08:03.670804 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:03.670808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:03.670825 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:08:03.670878 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:03.670881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:03.670952 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:03.671042 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:03.672284 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.673063 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.674293 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:03.674691 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:03.674786 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:03.677406 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:03.677423 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:03.677439 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:03.677477 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:03.680258 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:03.680999 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:03.681006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:03.681009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:03.681133 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:03.681455 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:03.681458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:03.681460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:03.681771 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:03.681982 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:03.682287 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 01:08:03.682458 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":263,"threads_started":1}
I20260929 01:08:03.683380 13306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:03.778280 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 13183
I20260929 01:08:03.795121 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 12518
I20260929 01:08:03.796247 12685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:03.914904 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 12518
I20260929 01:08:03.935144 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 12619
I20260929 01:08:03.936239 12873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:04.055478 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 12619
2026-09-29T01:08:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9617 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:08:04 dist-test-slave-41cl krb5kdc[13445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:08:04 dist-test-slave-41cl krb5kdc[13445](info): set up 2 sockets
Sep 29 01:08:04 dist-test-slave-41cl krb5kdc[13445](info): commencing operation
krb5kdc: starting...
W20260929 01:08:05.161353 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:08:05 dist-test-slave-41cl krb5kdc[13445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:08:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.254.keytab.
Entry for principal oryx/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.254.keytab.
I20260929 01:08:05.377652 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:38465
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.254.keytab
--principal=oryx/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:32797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:38465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:08:05.701920 13458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:05.702250 13458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:05.702373 13458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:05.709900 13458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:08:05.710007 13458 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 01:08:05.710047 13458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:05.710100 13458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:05.710148 13458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:08:05.710207 13458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:08:05.720108 13458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:32797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:38465
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.254.keytab
--principal=oryx/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:38465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:05.721390 13458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:05.722632 13458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:05.734412 13464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:05.734549 13466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:05.734440 13463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:05.735216 13458 server_base.cc:1034] running on GCE node
I20260929 01:08:05.735983 13458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:05.737459 13458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:05.738726 13458 hybrid_clock.cc:648] HybridClock initialized: now 1790644085738650 us; error 52 us; skew 500 ppm
I20260929 01:08:05.739065 13458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:05 dist-test-slave-41cl krb5kdc[13445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:05.742157 13458 init.cc:377] Logged in from keytab as oryx/127.5.198.254@KRBTEST.COM (short username oryx)
I20260929 01:08:05.743499 13458 webserver.cc:460] Webserver started at http://127.5.198.254:41297/ using document root <none> and password file <none>
I20260929 01:08:05.744169 13458 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:05.744309 13458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:05.744697 13458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:05.747545 13458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "c1e376999cce41c2831666404ecdbef1"
format_stamp: "Formatted at 2026-09-29 01:08:05 on dist-test-slave-41cl"
server_key: "b97686baa1aa089ed80ff911f96efdaf"
server_key_iv: "0e7d9f2eadce6874a57fdcb38c2d1065"
server_key_version: "encryptionkey@0"
I20260929 01:08:05.748288 13458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "c1e376999cce41c2831666404ecdbef1"
format_stamp: "Formatted at 2026-09-29 01:08:05 on dist-test-slave-41cl"
server_key: "b97686baa1aa089ed80ff911f96efdaf"
server_key_iv: "0e7d9f2eadce6874a57fdcb38c2d1065"
server_key_version: "encryptionkey@0"
I20260929 01:08:05.753019 13458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:08:05.756307 13473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:05.757797 13458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:08:05.758041 13458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "c1e376999cce41c2831666404ecdbef1"
format_stamp: "Formatted at 2026-09-29 01:08:05 on dist-test-slave-41cl"
server_key: "b97686baa1aa089ed80ff911f96efdaf"
server_key_iv: "0e7d9f2eadce6874a57fdcb38c2d1065"
server_key_version: "encryptionkey@0"
I20260929 01:08:05.758255 13458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:05.773945 13458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:05.778120 13458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:05.778374 13458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:05.798985 13458 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:38465
I20260929 01:08:05.799005 13534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:38465 every 8 connection(s)
I20260929 01:08:05.800243 13458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:08:05.803799 13535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:05.808307 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 13458
I20260929 01:08:05.808566 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:08:05.809203 5915 external_mini_cluster.cc:1428] Setting key 935cac908b8022b4f225d33bd344d785
I20260929 01:08:05.814512 13535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1: Bootstrap starting.
I20260929 01:08:05.818089 13535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:05.819238 13535 log.cc:824] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1: Log is configured to *not* fsync() on all Append() calls
I20260929 01:08:05.821897 13535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1: No bootstrap required, opened a new log
Sep 29 01:08:05 dist-test-slave-41cl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM
I20260929 01:08:05.832072 13538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:05.810990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.5.198.254:38465)
0929 01:08:05.811745 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:05.811757 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:05.811798 (+ 41us) server_negotiation.cc:407] Connection header received
0929 01:08:05.813460 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:05.813481 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:05.813835 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:05.814135 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:05.815822 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:05.817213 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:05.818991 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:05.819786 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:05.820010 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:05.823639 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:05.823651 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:05.823673 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:05.823713 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:05.827731 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:05.828396 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:05.828404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:05.828413 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:05.828658 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:05.829090 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:05.829095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:05.829098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:05.829732 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:05.830019 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:05.830413 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 01:08:05.830820 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":351,"threads_started":1}
I20260929 01:08:05.832042 13535 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e376999cce41c2831666404ecdbef1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38465 } }
I20260929 01:08:05.832455 13535 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:05.832587 13535 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e376999cce41c2831666404ecdbef1, State: Initialized, Role: FOLLOWER
I20260929 01:08:05.833209 13535 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e376999cce41c2831666404ecdbef1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38465 } }
I20260929 01:08:05.833496 13535 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:08:05.833635 13535 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:08:05.833820 13535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:05.836138 13535 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e376999cce41c2831666404ecdbef1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38465 } }
I20260929 01:08:05.836678 13535 leader_election.cc:302] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1e376999cce41c2831666404ecdbef1; no voters:
I20260929 01:08:05.837226 13535 leader_election.cc:288] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:08:05.837379 13540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:08:05.838366 13540 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [term 1 LEADER]: Becoming Leader. State: Replica: c1e376999cce41c2831666404ecdbef1, State: Running, Role: LEADER
I20260929 01:08:05.838889 13540 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e376999cce41c2831666404ecdbef1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38465 } }
I20260929 01:08:05.839004 13535 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:08:05.841511 13542 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1e376999cce41c2831666404ecdbef1. Latest consensus state: current_term: 1 leader_uuid: "c1e376999cce41c2831666404ecdbef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e376999cce41c2831666404ecdbef1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38465 } } }
I20260929 01:08:05.841869 13542 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:05.842463 13541 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1e376999cce41c2831666404ecdbef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e376999cce41c2831666404ecdbef1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38465 } } }
I20260929 01:08:05.842722 13546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:08:05.842727 13541 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1 [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:05.856195 13546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:08:05.867120 13546 catalog_manager.cc:1347] Generated new cluster ID: 3a10dfb8a1d1471c9533f40ad883bf68
I20260929 01:08:05.867230 13546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:08:05.878707 13546 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:08:05.880582 13546 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:08:05.890512 13546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1: Generated new TSK 0
I20260929 01:08:05.891317 13546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.193.keytab.
Entry for principal oryx/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.193.keytab.
I20260929 01:08:05.956305 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:38465
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.193.keytab
--principal=oryx/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:32797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:08:06.303411 13562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:06.303666 13562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:06.303757 13562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:06.310988 13562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:06.311089 13562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:06.311172 13562 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:08:06.311257 13562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:08:06.321144 13562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:32797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.193.keytab
--principal=oryx/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.198.254:38465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:06.322350 13562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:06.323653 13562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:06.337414 13568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:06.337446 13567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:06.337373 13570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:06.338152 13562 server_base.cc:1034] running on GCE node
I20260929 01:08:06.338734 13562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:06.339694 13562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:06.340950 13562 hybrid_clock.cc:648] HybridClock initialized: now 1790644086340867 us; error 58 us; skew 500 ppm
I20260929 01:08:06.341223 13562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:06 dist-test-slave-41cl krb5kdc[13445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644086, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:06.344584 13562 init.cc:377] Logged in from keytab as oryx/127.5.198.193@KRBTEST.COM (short username oryx)
I20260929 01:08:06.346004 13562 webserver.cc:460] Webserver started at http://127.5.198.193:37925/ using document root <none> and password file <none>
I20260929 01:08:06.346649 13562 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:06.346733 13562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:06.346984 13562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:06.349920 13562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "f1414d21b20f4c1aa99853821d1e0d21"
format_stamp: "Formatted at 2026-09-29 01:08:06 on dist-test-slave-41cl"
server_key: "868b6404bf8e7f8ebf80e1aa8d5c470d"
server_key_iv: "15f2ff82cd7076c18ed343ca064a3c89"
server_key_version: "encryptionkey@0"
I20260929 01:08:06.350649 13562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "f1414d21b20f4c1aa99853821d1e0d21"
format_stamp: "Formatted at 2026-09-29 01:08:06 on dist-test-slave-41cl"
server_key: "868b6404bf8e7f8ebf80e1aa8d5c470d"
server_key_iv: "15f2ff82cd7076c18ed343ca064a3c89"
server_key_version: "encryptionkey@0"
I20260929 01:08:06.356122 13562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:08:06.360023 13577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:06.361505 13562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:08:06.361732 13562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "f1414d21b20f4c1aa99853821d1e0d21"
format_stamp: "Formatted at 2026-09-29 01:08:06 on dist-test-slave-41cl"
server_key: "868b6404bf8e7f8ebf80e1aa8d5c470d"
server_key_iv: "15f2ff82cd7076c18ed343ca064a3c89"
server_key_version: "encryptionkey@0"
I20260929 01:08:06.361948 13562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:06.387971 13562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:06.392323 13562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:06.392647 13562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:06.395155 13562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:06.395237 13562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:06.395402 13562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:06.395462 13562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:08:06 dist-test-slave-41cl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644086, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM
I20260929 01:08:06.418687 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:06.395983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:50785 (local address 127.5.198.254:38465)
0929 01:08:06.396448 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:06.396454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:06.397742 (+ 1288us) server_negotiation.cc:407] Connection header received
0929 01:08:06.400887 (+ 3145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:06.400896 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:06.400998 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:06.401146 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:06.403345 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:06.404651 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:06.406565 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:06.407178 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:06.407327 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:06.411806 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:06.411812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:06.411815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:06.411849 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:06.415054 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:06.416041 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:06.416049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:06.416052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:06.416169 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:06.416840 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:06.416846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:06.416849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:06.417203 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:06.417360 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:06.418116 (+ 756us) server_negotiation.cc:299] Negotiation successful
0929 01:08:06.418326 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":210,"threads_started":1}
I20260929 01:08:06.419613 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:06.396673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38465 (local address 127.5.198.193:50785)
0929 01:08:06.397555 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:06.397616 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:06.400448 (+ 2832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:06.401450 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:06.401470 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:06.402282 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:06.403075 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:06.403098 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:06.404807 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:06.404813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:06.406404 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:06.406416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:06.407439 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:06.407446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:06.407653 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:06.408529 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:06.408569 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:06.411576 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:06.415240 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:06.415248 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:06.415262 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:06.415911 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:06.416295 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:06.416302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:06.416307 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:06.416710 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:06.417419 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:06.417424 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:06.417737 (+ 313us) client_negotiation.cc:769] Sending connection context
0929 01:08:06.418011 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 01:08:06.418490 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":336,"threads_started":1}
I20260929 01:08:06.440853 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:06.426040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:34579 (local address 127.5.198.254:38465)
0929 01:08:06.426200 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:06.426204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:06.426894 (+ 690us) server_negotiation.cc:407] Connection header received
0929 01:08:06.427092 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:06.427096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:06.427152 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:06.427250 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:06.428118 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:06.428904 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:06.430544 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:06.431210 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:06.431350 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:06.433844 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:06.433849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:06.433852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:06.433895 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:06.437537 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:06.438580 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:06.438588 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:06.438592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:06.438757 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:06.439246 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:06.439253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:06.439259 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:06.439568 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:06.439698 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:06.440240 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 01:08:06.440387 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:08:06.441107 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:06.426219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38465 (local address 127.5.198.193:34579)
0929 01:08:06.426785 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:06.426801 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:06.426956 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:06.427271 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:06.427275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:06.427709 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:06.427979 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:06.427988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:06.429096 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:06.429101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:06.430363 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:06.430375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:06.431427 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:06.431441 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:06.431711 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:06.432411 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:06.432443 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:06.433653 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:06.437714 (+ 4061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:06.437722 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:06.437729 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:06.438445 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:06.438882 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:06.438888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:06.438892 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:06.439071 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:06.439686 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:06.439694 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:06.439960 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 01:08:06.440172 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 01:08:06.440635 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":306,"threads_started":1}
I20260929 01:08:06.448983 13562 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:36481
I20260929 01:08:06.449026 13697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:36481 every 8 connection(s)
I20260929 01:08:06.450239 13562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:08:06.452360 13698 heartbeater.cc:347] Connected to a master server at 127.5.198.254:38465
I20260929 01:08:06.452598 13698 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:06.453243 13698 heartbeater.cc:511] Master 127.5.198.254:38465 requested a full tablet report, sending...
I20260929 01:08:06.455405 13490 ts_manager.cc:194] Registered new tserver with Master: f1414d21b20f4c1aa99853821d1e0d21 (127.5.198.193:36481)
I20260929 01:08:06.456979 13490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:50785
I20260929 01:08:06.459698 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 13562
I20260929 01:08:06.459882 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:08:06.460556 5915 external_mini_cluster.cc:1428] Setting key aca14e2e95a455a495aacb80a7766d27
WARNING: no policy specified for oryx/127.5.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.194.keytab.
Entry for principal oryx/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.194.keytab.
I20260929 01:08:06.504868 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:38465
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.194.keytab
--principal=oryx/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:32797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:08:06.824379 13702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:06.824746 13702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:06.824922 13702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:06.832037 13702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:06.832168 13702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:06.832324 13702 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:08:06.832502 13702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:08:06.842773 13702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:32797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.194.keytab
--principal=oryx/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.198.254:38465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:06.844094 13702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:06.845438 13702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:06.859113 13707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:06.859163 13708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:06.859294 13710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:06.860081 13702 server_base.cc:1034] running on GCE node
I20260929 01:08:06.860864 13702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:06.861835 13702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:06.863092 13702 hybrid_clock.cc:648] HybridClock initialized: now 1790644086863010 us; error 52 us; skew 500 ppm
I20260929 01:08:06.863444 13702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:06 dist-test-slave-41cl krb5kdc[13445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644086, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:06.867152 13702 init.cc:377] Logged in from keytab as oryx/127.5.198.194@KRBTEST.COM (short username oryx)
I20260929 01:08:06.868788 13702 webserver.cc:460] Webserver started at http://127.5.198.194:46861/ using document root <none> and password file <none>
I20260929 01:08:06.869517 13702 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:06.869653 13702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:06.869976 13702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:06.873422 13702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "de8e6eb4bdc8453ba8eb035d8058fa13"
format_stamp: "Formatted at 2026-09-29 01:08:06 on dist-test-slave-41cl"
server_key: "84da472f3c90245f2c9e4dd83895b7ec"
server_key_iv: "dbde8ce49fdcc3a09f57a62ab1e9c17e"
server_key_version: "encryptionkey@0"
I20260929 01:08:06.874228 13702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "de8e6eb4bdc8453ba8eb035d8058fa13"
format_stamp: "Formatted at 2026-09-29 01:08:06 on dist-test-slave-41cl"
server_key: "84da472f3c90245f2c9e4dd83895b7ec"
server_key_iv: "dbde8ce49fdcc3a09f57a62ab1e9c17e"
server_key_version: "encryptionkey@0"
I20260929 01:08:06.879712 13702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 01:08:06.883556 13717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:06.885174 13702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:08:06.885360 13702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "de8e6eb4bdc8453ba8eb035d8058fa13"
format_stamp: "Formatted at 2026-09-29 01:08:06 on dist-test-slave-41cl"
server_key: "84da472f3c90245f2c9e4dd83895b7ec"
server_key_iv: "dbde8ce49fdcc3a09f57a62ab1e9c17e"
server_key_version: "encryptionkey@0"
I20260929 01:08:06.885612 13702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:06.900045 13702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:06.905030 13702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:06.905339 13702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:06.908146 13702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:06.908248 13702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:06.908368 13702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:06.908478 13702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:08:06 dist-test-slave-41cl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644086, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM
I20260929 01:08:06.933230 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:06.908739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:52905 (local address 127.5.198.254:38465)
0929 01:08:06.908953 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:06.908960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:06.910427 (+ 1467us) server_negotiation.cc:407] Connection header received
0929 01:08:06.912730 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:06.912737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:06.912835 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:06.912967 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:06.915364 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:06.916796 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:06.919424 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:06.920119 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:06.920252 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:06.925850 (+ 5598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:06.925856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:06.925859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:06.925894 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:06.928797 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:06.930292 (+ 1495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:06.930299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:06.930302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:06.930463 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:06.931359 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:06.931362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:06.931364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:06.931624 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:06.931811 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:06.932667 (+ 856us) server_negotiation.cc:299] Negotiation successful
0929 01:08:06.932882 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 01:08:06.934854 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:06.909392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38465 (local address 127.5.198.194:52905)
0929 01:08:06.910237 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:06.910277 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:06.912382 (+ 2105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:06.913239 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:06.913252 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:06.914165 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:06.915080 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:06.915099 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:06.917000 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:06.917009 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:06.919224 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:06.919237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:06.920365 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:06.920372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:06.920667 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:06.921691 (+ 1024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:06.921716 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:06.925554 (+ 3838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:06.929041 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:06.929050 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:06.929068 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:06.930122 (+ 1054us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:06.930610 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:06.930615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:06.930618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:06.930935 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:06.931783 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:06.931792 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:06.932246 (+ 454us) client_negotiation.cc:769] Sending connection context
0929 01:08:06.932579 (+ 333us) client_negotiation.cc:240] Negotiation successful
0929 01:08:06.933134 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":359,"threads_started":1}
I20260929 01:08:06.955143 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:06.941298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38465 (local address 127.5.198.194:48947)
0929 01:08:06.941919 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:06.941935 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:06.942081 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:06.942402 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:06.942407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:06.942868 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:06.943135 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:06.943143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:06.944133 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:06.944138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:06.945664 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:06.945678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:06.946391 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:06.946397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:06.946562 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:06.947176 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:06.947192 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:06.947847 (+ 655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:06.951360 (+ 3513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:06.951366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:06.951370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:06.951995 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:06.952521 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:06.952530 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:06.952534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:06.952657 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:06.953575 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:06.953583 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:06.953845 (+ 262us) client_negotiation.cc:769] Sending connection context
0929 01:08:06.954062 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 01:08:06.954481 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":354,"threads_started":1}
I20260929 01:08:06.956501 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:06.940988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:48947 (local address 127.5.198.254:38465)
0929 01:08:06.941185 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:06.941189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:06.942018 (+ 829us) server_negotiation.cc:407] Connection header received
0929 01:08:06.942213 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:06.942223 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:06.942282 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:06.942382 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:06.943271 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:06.943989 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:06.945808 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:06.946204 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:06.946352 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:06.948030 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:06.948036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:06.948039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:06.948074 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:06.951185 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:06.952191 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:06.952198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:06.952204 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:06.952378 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:06.952828 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:06.952837 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:06.952843 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:06.953310 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:06.953501 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:06.955706 (+ 2205us) server_negotiation.cc:299] Negotiation successful
0929 01:08:06.955990 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:08:06.966176 13702 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:45535
I20260929 01:08:06.966310 13836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:45535 every 8 connection(s)
I20260929 01:08:06.967410 13702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:08:06.969594 13837 heartbeater.cc:347] Connected to a master server at 127.5.198.254:38465
I20260929 01:08:06.969851 13837 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:06.970541 13837 heartbeater.cc:511] Master 127.5.198.254:38465 requested a full tablet report, sending...
I20260929 01:08:06.971923 13490 ts_manager.cc:194] Registered new tserver with Master: de8e6eb4bdc8453ba8eb035d8058fa13 (127.5.198.194:45535)
I20260929 01:08:06.972821 13490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:52905
I20260929 01:08:06.975454 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 13702
I20260929 01:08:06.975632 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:08:06.976140 5915 external_mini_cluster.cc:1428] Setting key aef06d0516ba0e7506b467f212bf9dc6
WARNING: no policy specified for oryx/127.5.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.195.keytab.
Entry for principal oryx/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.195.keytab.
I20260929 01:08:07.024362 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:38465
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.195.keytab
--principal=oryx/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:32797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:08:07.342927 13841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:07.343168 13841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:07.343319 13841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:07.350517 13841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:07.350620 13841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:07.350818 13841 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:08:07.350929 13841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:08:07.360944 13841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:32797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.195.keytab
--principal=oryx/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.198.254:38465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:07.362229 13841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:07.363585 13841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:07.377481 13846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:07.377426 13849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:07.377735 13847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:07.378724 13841 server_base.cc:1034] running on GCE node
I20260929 01:08:07.379336 13841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:07.380308 13841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:07.381510 13841 hybrid_clock.cc:648] HybridClock initialized: now 1790644087381448 us; error 77 us; skew 500 ppm
I20260929 01:08:07.381807 13841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:07 dist-test-slave-41cl krb5kdc[13445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644087, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:07.385008 13841 init.cc:377] Logged in from keytab as oryx/127.5.198.195@KRBTEST.COM (short username oryx)
I20260929 01:08:07.386519 13841 webserver.cc:460] Webserver started at http://127.5.198.195:46001/ using document root <none> and password file <none>
I20260929 01:08:07.387202 13841 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:07.387320 13841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:07.387646 13841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:07.390616 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "df78540f1e544a9da2dcc479af683031"
format_stamp: "Formatted at 2026-09-29 01:08:07 on dist-test-slave-41cl"
server_key: "2dadebf5e88b99c58d9b069ce9cc7392"
server_key_iv: "471a47de58843371f9ae3e6a64eeb061"
server_key_version: "encryptionkey@0"
I20260929 01:08:07.391438 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "df78540f1e544a9da2dcc479af683031"
format_stamp: "Formatted at 2026-09-29 01:08:07 on dist-test-slave-41cl"
server_key: "2dadebf5e88b99c58d9b069ce9cc7392"
server_key_iv: "471a47de58843371f9ae3e6a64eeb061"
server_key_version: "encryptionkey@0"
I20260929 01:08:07.396610 13841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 01:08:07.400050 13856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:07.401448 13841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:08:07.401618 13841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "df78540f1e544a9da2dcc479af683031"
format_stamp: "Formatted at 2026-09-29 01:08:07 on dist-test-slave-41cl"
server_key: "2dadebf5e88b99c58d9b069ce9cc7392"
server_key_iv: "471a47de58843371f9ae3e6a64eeb061"
server_key_version: "encryptionkey@0"
I20260929 01:08:07.401842 13841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:07.414379 13841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:07.418520 13841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:07.418823 13841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:07.421518 13841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:07.421618 13841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:07.421777 13841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:07.421885 13841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 29 01:08:07 dist-test-slave-41cl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644087, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM
I20260929 01:08:07.445211 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:07.421977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:57227 (local address 127.5.198.254:38465)
0929 01:08:07.422180 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:07.422189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:07.423568 (+ 1379us) server_negotiation.cc:407] Connection header received
0929 01:08:07.425829 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:07.425835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:07.425923 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:07.426077 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:07.428738 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:07.429719 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:07.433008 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:07.433588 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:07.433738 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:07.439262 (+ 5524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:07.439269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:07.439273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:07.439318 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:07.442122 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:07.442846 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:07.442851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:07.442855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:07.443028 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:07.443454 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:07.443466 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:07.443470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:07.443730 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:07.443898 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:07.444644 (+ 746us) server_negotiation.cc:299] Negotiation successful
0929 01:08:07.444857 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 01:08:07.446197 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:07.422594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38465 (local address 127.5.198.195:57227)
0929 01:08:07.423408 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:07.423452 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:07.425532 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:07.426371 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:07.426390 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:07.427305 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:07.428400 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:07.428437 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:07.429923 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:07.429931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:07.432776 (+ 2845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:07.432796 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:07.433893 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:07.433908 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:07.434272 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:07.435067 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:07.435105 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:07.438998 (+ 3893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:07.442282 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:07.442291 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:07.442306 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:07.442717 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:07.443169 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:07.443173 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:07.443175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:07.443329 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:07.443859 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:07.443867 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:07.444214 (+ 347us) client_negotiation.cc:769] Sending connection context
0929 01:08:07.444467 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 01:08:07.445018 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":409,"threads_started":1}
I20260929 01:08:07.459587 13698 heartbeater.cc:503] Master 127.5.198.254:38465 was elected leader, sending a full tablet report...
I20260929 01:08:07.471589 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:07.457034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:42743 (local address 127.5.198.254:38465)
0929 01:08:07.457199 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:07.457206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:07.457640 (+ 434us) server_negotiation.cc:407] Connection header received
0929 01:08:07.457837 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:07.457843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:07.457931 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:07.458053 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:07.459957 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:07.460884 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:07.462796 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:07.463246 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:07.463404 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:07.465563 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:07.465569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:07.465574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:07.465612 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:07.469111 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:07.469771 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:07.469777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:07.469780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:07.469903 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:07.470249 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:07.470255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:07.470258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:07.470507 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:07.470653 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:07.471028 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 01:08:07.471228 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 01:08:07.471676 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:07.456909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38465 (local address 127.5.198.195:42743)
0929 01:08:07.457534 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:07.457550 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:07.457705 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:07.458142 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:07.458148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:07.459164 (+ 1016us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:07.459744 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:07.459764 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:07.461086 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:07.461094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:07.462628 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:07.462639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:07.463453 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:07.463460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:07.463643 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:07.464273 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:07.464289 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:07.465310 (+ 1021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:07.469238 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:07.469245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:07.469249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:07.469651 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:07.470016 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:07.470021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:07.470024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:07.470145 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:07.470619 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:07.470624 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:07.470797 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 01:08:07.470932 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 01:08:07.471228 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":386,"threads_started":1}
I20260929 01:08:07.486502 13841 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:44017
I20260929 01:08:07.486523 13975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:44017 every 8 connection(s)
I20260929 01:08:07.487767 13841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:08:07.490041 13976 heartbeater.cc:347] Connected to a master server at 127.5.198.254:38465
I20260929 01:08:07.490288 13976 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:07.490942 13976 heartbeater.cc:511] Master 127.5.198.254:38465 requested a full tablet report, sending...
I20260929 01:08:07.492282 13490 ts_manager.cc:194] Registered new tserver with Master: df78540f1e544a9da2dcc479af683031 (127.5.198.195:44017)
I20260929 01:08:07.493155 13490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:57227
I20260929 01:08:07.494199 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 13841
I20260929 01:08:07.494431 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:08:07.494884 5915 external_mini_cluster.cc:1428] Setting key 0787c1dfc2a1b3efa7b12cb6c3e659b8
I20260929 01:08:07.497697 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 01:08:07 dist-test-slave-41cl krb5kdc[13445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 01:08:07 dist-test-slave-41cl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:07 dist-test-slave-41cl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:07.975140 13837 heartbeater.cc:503] Master 127.5.198.254:38465 was elected leader, sending a full tablet report...
I20260929 01:08:08.495594 13976 heartbeater.cc:503] Master 127.5.198.254:38465 was elected leader, sending a full tablet report...
W20260929 01:08:08.619438 13984 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.078s user 0.000s sys 0.015s
W20260929 01:08:08.620630 13586 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:07.535039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53932 (local address 127.5.198.254:38465)
0929 01:08:07.535299 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:07.535306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:07.535498 (+ 192us) server_negotiation.cc:407] Connection header received
0929 01:08:07.535641 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:07.535645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:07.535708 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:07.535803 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:07.537253 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:07.538111 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:07.539523 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:07.539976 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:07.540079 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:08.620340 (+1080261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53932: BlockingRecv error: recv got EOF from 127.0.0.1:53932 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 29 01:08:08 dist-test-slave-41cl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM
I20260929 01:08:08.648417 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:08.629687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.5.198.254:38465)
0929 01:08:08.629974 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:08.629981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:08.630162 (+ 181us) server_negotiation.cc:407] Connection header received
0929 01:08:08.630327 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:08.630331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:08.630435 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:08.630521 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:08.631814 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.632666 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.633946 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.634365 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.634468 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:08.643270 (+ 8802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:08.643276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:08.643280 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:08.643333 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:08.646063 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:08.646698 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:08.646702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:08.646704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:08.646813 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:08.647132 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:08.647135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:08.647137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:08.647391 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:08.647597 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:08.647902 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 01:08:08.648068 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 01:08:08.652716 13490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:08:08.659701 13490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:08:08.693176 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:08.671854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.5.198.193:36481)
0929 01:08:08.683908 (+ 12054us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:08.683916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:08.683944 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:08:08.684061 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:08.684072 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:08.684410 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:08.684884 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:08.686021 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.688118 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.690997 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.691901 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.692049 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:08.692268 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:08.692656 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 01:08:08.692842 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11869,"thread_start_us":12636,"threads_started":1}
I20260929 01:08:08.693967 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:08.671631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:36481 (local address 127.0.0.1:37234)
0929 01:08:08.683646 (+ 12015us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:08.683665 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:08.683773 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:08.685171 (+ 1398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:08.685183 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:08.685224 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:08.685746 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:08.685762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.688868 (+ 3106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.688872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:08.690789 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:08.690812 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.692107 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.692113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:08.692283 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:08.692351 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 01:08:08.693376 (+ 1025us) client_negotiation.cc:240] Negotiation successful
0929 01:08:08.693562 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11873,"thread_start_us":10677,"threads_started":1}
I20260929 01:08:08.696005 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:08.671285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:44017 (local address 127.0.0.1:46694)
0929 01:08:08.682888 (+ 11603us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:08.682946 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:08.683178 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:08.686290 (+ 3112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:08.686297 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:08.686331 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:08.686847 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:08.686863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.689118 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.689127 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:08.692072 (+ 2945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:08.692095 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.695224 (+ 3129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.695231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:08.695410 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:08.695442 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 01:08:08.695518 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 01:08:08.695635 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11235,"thread_start_us":12090,"threads_started":1}
I20260929 01:08:08.696521 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:08.671506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46694 (local address 127.5.198.195:44017)
0929 01:08:08.685473 (+ 13967us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:08.685481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:08.685506 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:08:08.685589 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:08.685596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:08.685834 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:08.686043 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:08.687063 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.688914 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.692310 (+ 3396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.693936 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.694076 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:08.695942 (+ 1866us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:08.696026 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 01:08:08.696195 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13821,"thread_start_us":13703,"threads_started":1}
I20260929 01:08:08.697851 13628 tablet_service.cc:1467] Processing CreateTablet for tablet 9703ab1bedab4db79bbbc9ae994ab760 (DEFAULT_TABLE table=test-table [id=18e6fa2a8d8c4899b897ef1469b2c4f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:08.699187 13628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9703ab1bedab4db79bbbc9ae994ab760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:08.699595 13906 tablet_service.cc:1467] Processing CreateTablet for tablet 9703ab1bedab4db79bbbc9ae994ab760 (DEFAULT_TABLE table=test-table [id=18e6fa2a8d8c4899b897ef1469b2c4f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:08.700742 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:08.685543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:45535 (local address 127.0.0.1:52984)
0929 01:08:08.692728 (+ 7185us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:08.692745 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:08.692855 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:08.693705 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:08.693710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:08.693734 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:08.694140 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:08.694152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.695898 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.695906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:08.697887 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:08.697901 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.699941 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.699947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:08.700118 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:08.700153 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 01:08:08.700244 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 01:08:08.700366 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7078,"thread_start_us":6924,"threads_started":1}
I20260929 01:08:08.700961 13906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9703ab1bedab4db79bbbc9ae994ab760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:08.701220 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:08.684055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52984 (local address 127.5.198.194:45535)
0929 01:08:08.688783 (+ 4728us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:08.688791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:08.692861 (+ 4070us) server_negotiation.cc:407] Connection header received
0929 01:08:08.692966 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:08.692973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:08.693197 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:08.693501 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:08.694344 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.695744 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.698063 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.699444 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.699586 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:08.699785 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:08.700631 (+ 846us) server_negotiation.cc:299] Negotiation successful
0929 01:08:08.700804 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4591,"thread_start_us":4472,"threads_started":1}
I20260929 01:08:08.704586 13767 tablet_service.cc:1467] Processing CreateTablet for tablet 9703ab1bedab4db79bbbc9ae994ab760 (DEFAULT_TABLE table=test-table [id=18e6fa2a8d8c4899b897ef1469b2c4f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:08.705850 13767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9703ab1bedab4db79bbbc9ae994ab760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:08.711201 14001 tablet_bootstrap.cc:492] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21: Bootstrap starting.
I20260929 01:08:08.714177 14001 tablet_bootstrap.cc:654] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:08.714936 14002 tablet_bootstrap.cc:492] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031: Bootstrap starting.
I20260929 01:08:08.716511 14001 log.cc:824] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21: Log is configured to *not* fsync() on all Append() calls
I20260929 01:08:08.718348 14002 tablet_bootstrap.cc:654] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:08.718991 14004 tablet_bootstrap.cc:492] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13: Bootstrap starting.
I20260929 01:08:08.719555 14001 tablet_bootstrap.cc:492] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21: No bootstrap required, opened a new log
I20260929 01:08:08.719573 14002 log.cc:824] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031: Log is configured to *not* fsync() on all Append() calls
I20260929 01:08:08.719825 14001 ts_tablet_manager.cc:1397] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 01:08:08.722579 14004 tablet_bootstrap.cc:654] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:08.722779 14002 tablet_bootstrap.cc:492] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031: No bootstrap required, opened a new log
I20260929 01:08:08.723042 14002 ts_tablet_manager.cc:1397] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 01:08:08.723613 14004 log.cc:824] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13: Log is configured to *not* fsync() on all Append() calls
I20260929 01:08:08.726132 14004 tablet_bootstrap.cc:492] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13: No bootstrap required, opened a new log
I20260929 01:08:08.726380 14004 ts_tablet_manager.cc:1397] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260929 01:08:08.730811 14001 raft_consensus.cc:348] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } }
I20260929 01:08:08.731283 14001 raft_consensus.cc:374] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:08.731400 14001 raft_consensus.cc:729] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1414d21b20f4c1aa99853821d1e0d21, State: Initialized, Role: FOLLOWER
I20260929 01:08:08.732028 14001 consensus_queue.cc:260] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } }
I20260929 01:08:08.733299 14002 raft_consensus.cc:348] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } }
I20260929 01:08:08.733722 14002 raft_consensus.cc:374] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:08.733798 14001 ts_tablet_manager.cc:1428] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260929 01:08:08.733821 14002 raft_consensus.cc:729] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df78540f1e544a9da2dcc479af683031, State: Initialized, Role: FOLLOWER
I20260929 01:08:08.734385 14002 consensus_queue.cc:260] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } }
I20260929 01:08:08.734252 14004 raft_consensus.cc:348] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } }
I20260929 01:08:08.734649 14004 raft_consensus.cc:374] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:08.734717 14004 raft_consensus.cc:729] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de8e6eb4bdc8453ba8eb035d8058fa13, State: Initialized, Role: FOLLOWER
I20260929 01:08:08.735396 14004 consensus_queue.cc:260] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } }
I20260929 01:08:08.735935 14002 ts_tablet_manager.cc:1428] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 01:08:08.738260 14004 ts_tablet_manager.cc:1428] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260929 01:08:08.740305 13977 tablet.cc:2369] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:08:08.802716 14008 raft_consensus.cc:482] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:08:08.803042 14008 raft_consensus.cc:504] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } }
I20260929 01:08:08.804816 14008 leader_election.cc:288] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1414d21b20f4c1aa99853821d1e0d21 (127.5.198.193:36481), de8e6eb4bdc8453ba8eb035d8058fa13 (127.5.198.194:45535)
I20260929 01:08:08.812372 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:08.805364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:45307 (local address 127.5.198.194:45535)
0929 01:08:08.805612 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:08.805622 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:08.806753 (+ 1131us) server_negotiation.cc:407] Connection header received
0929 01:08:08.807272 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:08.807278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:08.807366 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:08.807448 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:08.808011 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.808741 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.810615 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.811502 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.811661 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:08.811732 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:08.812015 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 01:08:08.812140 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 01:08:08.812729 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:08.805238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:45535 (local address 127.5.198.195:45307)
0929 01:08:08.806659 (+ 1421us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:08.806674 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:08.806778 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:08.807503 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:08.807508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:08.807533 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:08.807868 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:08.807878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.808881 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.808886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:08.810470 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:08.810481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.811695 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.811701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:08.811847 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:08.811883 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 01:08:08.811979 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 01:08:08.812104 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1317,"mutex_wait_us":89,"thread_start_us":767,"threads_started":1}
I20260929 01:08:08.813524 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:08.805575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:48101 (local address 127.5.198.193:36481)
0929 01:08:08.805821 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:08.805831 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:08.806573 (+ 742us) server_negotiation.cc:407] Connection header received
0929 01:08:08.806896 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:08.806901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:08.806989 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:08.807084 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:08.807615 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.808740 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.811282 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.812383 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.812586 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:08.812668 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:08.813171 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 01:08:08.813302 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 01:08:08.813603 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:08.805007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:36481 (local address 127.5.198.195:48101)
0929 01:08:08.806455 (+ 1448us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:08.806483 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:08.806694 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:08.807110 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:08.807114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:08.807134 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:08.807495 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:08.807505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.808919 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.808925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:08.811092 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:08.811106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.812765 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.812769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:08.812959 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:08.812992 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:08:08.813119 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 01:08:08.813237 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1268,"mutex_wait_us":26,"thread_start_us":1384,"threads_started":1}
I20260929 01:08:08.814136 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9703ab1bedab4db79bbbc9ae994ab760" candidate_uuid: "df78540f1e544a9da2dcc479af683031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" is_pre_election: true
I20260929 01:08:08.814641 13792 raft_consensus.cc:2454] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df78540f1e544a9da2dcc479af683031 in term 0.
I20260929 01:08:08.815286 13858 leader_election.cc:302] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de8e6eb4bdc8453ba8eb035d8058fa13, df78540f1e544a9da2dcc479af683031; no voters:
I20260929 01:08:08.815179 13648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9703ab1bedab4db79bbbc9ae994ab760" candidate_uuid: "df78540f1e544a9da2dcc479af683031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1414d21b20f4c1aa99853821d1e0d21" is_pre_election: true
I20260929 01:08:08.815649 14008 raft_consensus.cc:2781] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:08:08.815670 13648 raft_consensus.cc:2454] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df78540f1e544a9da2dcc479af683031 in term 0.
I20260929 01:08:08.815800 14008 raft_consensus.cc:482] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:08:08.815922 14008 raft_consensus.cc:3037] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:08.818125 14008 raft_consensus.cc:504] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } }
I20260929 01:08:08.818645 14008 leader_election.cc:288] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [CANDIDATE]: Term 1 election: Requested vote from peers f1414d21b20f4c1aa99853821d1e0d21 (127.5.198.193:36481), de8e6eb4bdc8453ba8eb035d8058fa13 (127.5.198.194:45535)
I20260929 01:08:08.819176 13648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9703ab1bedab4db79bbbc9ae994ab760" candidate_uuid: "df78540f1e544a9da2dcc479af683031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1414d21b20f4c1aa99853821d1e0d21"
I20260929 01:08:08.819176 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9703ab1bedab4db79bbbc9ae994ab760" candidate_uuid: "df78540f1e544a9da2dcc479af683031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13"
I20260929 01:08:08.819386 13792 raft_consensus.cc:3037] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:08.819389 13648 raft_consensus.cc:3037] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:08.822631 13648 raft_consensus.cc:2454] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df78540f1e544a9da2dcc479af683031 in term 1.
I20260929 01:08:08.822631 13792 raft_consensus.cc:2454] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df78540f1e544a9da2dcc479af683031 in term 1.
I20260929 01:08:08.823010 13857 leader_election.cc:302] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df78540f1e544a9da2dcc479af683031, f1414d21b20f4c1aa99853821d1e0d21; no voters:
I20260929 01:08:08.823351 14008 raft_consensus.cc:2781] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:08:08.823758 14008 raft_consensus.cc:686] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [term 1 LEADER]: Becoming Leader. State: Replica: df78540f1e544a9da2dcc479af683031, State: Running, Role: LEADER
I20260929 01:08:08.824244 14008 consensus_queue.cc:237] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } }
I20260929 01:08:08.828832 13488 catalog_manager.cc:5495] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 reported cstate change: term changed from 0 to 1, leader changed from <none> to df78540f1e544a9da2dcc479af683031 (127.5.198.195). New cstate: current_term: 1 leader_uuid: "df78540f1e544a9da2dcc479af683031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } health_report { overall_health: UNKNOWN } } }
I20260929 01:08:08.879287 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:08.874207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.5.198.195:44017)
0929 01:08:08.874398 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:08.874403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:08.874421 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:08:08.874546 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:08.874549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:08.874638 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:08.874762 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:08:08.875084 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.875815 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.877189 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:08.877591 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:08.877685 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:08.878204 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:08:08.878763 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:08:08.878910 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:08.879023 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 01:08:08.879088 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:08:08.887313 13648 raft_consensus.cc:1261] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Refusing update from remote peer df78540f1e544a9da2dcc479af683031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:08:08.887341 13792 raft_consensus.cc:1261] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 FOLLOWER]: Refusing update from remote peer df78540f1e544a9da2dcc479af683031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:08:08.888316 14012 consensus_queue.cc:1035] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:08:08.888789 14008 consensus_queue.cc:1035] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:08:08.909931 14017 mvcc.cc:205] Tried to move back new op lower bound from 7334478188073250816 to 7334478187828281344. Current Snapshot: MvccSnapshot[applied={T|T < 7334478188073250816 or (T in {7334478188073250816})}]
I20260929 01:08:08.913596 13489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:08:08.913882 13489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:08:08.917560 13489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1: Sending DeleteTablet for 3 replicas of tablet 9703ab1bedab4db79bbbc9ae994ab760
I20260929 01:08:08.918876 13628 tablet_service.cc:1514] Processing DeleteTablet for tablet 9703ab1bedab4db79bbbc9ae994ab760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:08 UTC) from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:37234
I20260929 01:08:08.919103 13906 tablet_service.cc:1514] Processing DeleteTablet for tablet 9703ab1bedab4db79bbbc9ae994ab760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:08 UTC) from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:46694
I20260929 01:08:08.919731 13767 tablet_service.cc:1514] Processing DeleteTablet for tablet 9703ab1bedab4db79bbbc9ae994ab760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:08 UTC) from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:52984
I20260929 01:08:08.921211 13488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:08:08.922153 13488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:08:08.929340 14027 tablet_replica.cc:330] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13: stopping tablet replica
I20260929 01:08:08.930682 14027 raft_consensus.cc:2229] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:08:08.931860 14027 raft_consensus.cc:2258] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:08.934362 13905 tablet_service.cc:1467] Processing CreateTablet for tablet bbaf5b62ef9c46ce8e938421f6ee0fc1 (DEFAULT_TABLE table=test-table [id=408e61a68e3a4fbf8eeaf64c4a04aad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:08.934362 13627 tablet_service.cc:1467] Processing CreateTablet for tablet bbaf5b62ef9c46ce8e938421f6ee0fc1 (DEFAULT_TABLE table=test-table [id=408e61a68e3a4fbf8eeaf64c4a04aad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:08.934999 13627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbaf5b62ef9c46ce8e938421f6ee0fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:08.935017 13905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbaf5b62ef9c46ce8e938421f6ee0fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:08.939405 13767 tablet_service.cc:1467] Processing CreateTablet for tablet bbaf5b62ef9c46ce8e938421f6ee0fc1 (DEFAULT_TABLE table=test-table [id=408e61a68e3a4fbf8eeaf64c4a04aad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:08.939891 13767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbaf5b62ef9c46ce8e938421f6ee0fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:08.947144 14001 tablet_bootstrap.cc:492] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21: Bootstrap starting.
I20260929 01:08:08.948983 14001 tablet_bootstrap.cc:654] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:08.950817 14002 tablet_bootstrap.cc:492] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031: Bootstrap starting.
I20260929 01:08:08.952997 14002 tablet_bootstrap.cc:654] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:08.957751 14004 tablet_bootstrap.cc:492] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13: Bootstrap starting.
I20260929 01:08:08.959615 14004 tablet_bootstrap.cc:654] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:08.961678 14001 tablet_bootstrap.cc:492] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21: No bootstrap required, opened a new log
I20260929 01:08:08.961855 14001 ts_tablet_manager.cc:1397] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260929 01:08:08.964294 14001 raft_consensus.cc:348] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:08.964622 14001 raft_consensus.cc:374] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:08.964723 14001 raft_consensus.cc:729] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1414d21b20f4c1aa99853821d1e0d21, State: Initialized, Role: FOLLOWER
I20260929 01:08:08.965066 14001 consensus_queue.cc:260] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:08.966069 14001 ts_tablet_manager.cc:1428] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 01:08:08.966813 14025 tablet_replica.cc:330] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21: stopping tablet replica
I20260929 01:08:08.967314 14025 raft_consensus.cc:2229] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:08:08.970399 14025 raft_consensus.cc:2258] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:08.972585 14025 ts_tablet_manager.cc:1905] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:08.981734 14002 tablet_bootstrap.cc:492] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031: No bootstrap required, opened a new log
I20260929 01:08:08.982861 14004 tablet_bootstrap.cc:492] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13: No bootstrap required, opened a new log
I20260929 01:08:08.983103 14004 ts_tablet_manager.cc:1397] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.005s
I20260929 01:08:08.983876 14025 ts_tablet_manager.cc:1918] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:08:08.984051 14025 log.cc:1196] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/9703ab1bedab4db79bbbc9ae994ab760
I20260929 01:08:08.984658 14025 ts_tablet_manager.cc:1939] T 9703ab1bedab4db79bbbc9ae994ab760 P f1414d21b20f4c1aa99853821d1e0d21: Deleting consensus metadata
I20260929 01:08:08.986028 14027 ts_tablet_manager.cc:1905] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:08.986296 14002 ts_tablet_manager.cc:1397] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.005s
I20260929 01:08:08.986526 13474 catalog_manager.cc:4841] TS f1414d21b20f4c1aa99853821d1e0d21 (127.5.198.193:36481): tablet 9703ab1bedab4db79bbbc9ae994ab760 (table test-table [id=18e6fa2a8d8c4899b897ef1469b2c4f1]) successfully deleted
I20260929 01:08:08.986775 14004 raft_consensus.cc:348] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:08.987079 14004 raft_consensus.cc:374] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:08.987185 14004 raft_consensus.cc:729] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de8e6eb4bdc8453ba8eb035d8058fa13, State: Initialized, Role: FOLLOWER
I20260929 01:08:08.987469 14004 consensus_queue.cc:260] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:08.988327 14002 raft_consensus.cc:348] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:08.989137 14004 ts_tablet_manager.cc:1428] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260929 01:08:08.989382 14002 raft_consensus.cc:374] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:08.989492 14002 raft_consensus.cc:729] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df78540f1e544a9da2dcc479af683031, State: Initialized, Role: FOLLOWER
I20260929 01:08:08.988754 14026 tablet_replica.cc:330] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031: stopping tablet replica
I20260929 01:08:08.990103 14026 raft_consensus.cc:2229] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:08:08.990767 14026 raft_consensus.cc:2258] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:08.992250 14002 consensus_queue.cc:260] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:08.995206 14002 ts_tablet_manager.cc:1428] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260929 01:08:08.994228 14026 ts_tablet_manager.cc:1905] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:08.997951 14027 ts_tablet_manager.cc:1918] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:08:08.998118 14027 log.cc:1196] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/9703ab1bedab4db79bbbc9ae994ab760
I20260929 01:08:08.998634 14027 ts_tablet_manager.cc:1939] T 9703ab1bedab4db79bbbc9ae994ab760 P de8e6eb4bdc8453ba8eb035d8058fa13: Deleting consensus metadata
I20260929 01:08:09.000324 13475 catalog_manager.cc:4841] TS de8e6eb4bdc8453ba8eb035d8058fa13 (127.5.198.194:45535): tablet 9703ab1bedab4db79bbbc9ae994ab760 (table test-table [id=18e6fa2a8d8c4899b897ef1469b2c4f1]) successfully deleted
I20260929 01:08:09.002660 14026 ts_tablet_manager.cc:1918] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:08:09.002848 14026 log.cc:1196] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/9703ab1bedab4db79bbbc9ae994ab760
I20260929 01:08:09.003365 14026 ts_tablet_manager.cc:1939] T 9703ab1bedab4db79bbbc9ae994ab760 P df78540f1e544a9da2dcc479af683031: Deleting consensus metadata
I20260929 01:08:09.004974 13475 catalog_manager.cc:4841] TS df78540f1e544a9da2dcc479af683031 (127.5.198.195:44017): tablet 9703ab1bedab4db79bbbc9ae994ab760 (table test-table [id=18e6fa2a8d8c4899b897ef1469b2c4f1]) successfully deleted
W20260929 01:08:09.205364 13699 tablet.cc:2369] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:08:09.211910 14012 raft_consensus.cc:482] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:08:09.212177 14012 raft_consensus.cc:504] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.213060 14012 leader_election.cc:288] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1414d21b20f4c1aa99853821d1e0d21 (127.5.198.193:36481), de8e6eb4bdc8453ba8eb035d8058fa13 (127.5.198.194:45535)
I20260929 01:08:09.213616 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbaf5b62ef9c46ce8e938421f6ee0fc1" candidate_uuid: "df78540f1e544a9da2dcc479af683031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" is_pre_election: true
I20260929 01:08:09.213657 13648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbaf5b62ef9c46ce8e938421f6ee0fc1" candidate_uuid: "df78540f1e544a9da2dcc479af683031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1414d21b20f4c1aa99853821d1e0d21" is_pre_election: true
I20260929 01:08:09.213860 13792 raft_consensus.cc:2454] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df78540f1e544a9da2dcc479af683031 in term 0.
I20260929 01:08:09.213929 13648 raft_consensus.cc:2454] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df78540f1e544a9da2dcc479af683031 in term 0.
I20260929 01:08:09.214326 13858 leader_election.cc:302] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de8e6eb4bdc8453ba8eb035d8058fa13, df78540f1e544a9da2dcc479af683031; no voters:
I20260929 01:08:09.214700 14012 raft_consensus.cc:2781] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:08:09.214810 14012 raft_consensus.cc:482] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:08:09.214991 14012 raft_consensus.cc:3037] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:09.217305 14012 raft_consensus.cc:504] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.217940 14012 leader_election.cc:288] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [CANDIDATE]: Term 1 election: Requested vote from peers f1414d21b20f4c1aa99853821d1e0d21 (127.5.198.193:36481), de8e6eb4bdc8453ba8eb035d8058fa13 (127.5.198.194:45535)
I20260929 01:08:09.218286 13648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbaf5b62ef9c46ce8e938421f6ee0fc1" candidate_uuid: "df78540f1e544a9da2dcc479af683031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1414d21b20f4c1aa99853821d1e0d21"
I20260929 01:08:09.218497 13648 raft_consensus.cc:3037] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:09.218467 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbaf5b62ef9c46ce8e938421f6ee0fc1" candidate_uuid: "df78540f1e544a9da2dcc479af683031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13"
I20260929 01:08:09.218649 13792 raft_consensus.cc:3037] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Advancing to term 1
W20260929 01:08:09.220564 13838 tablet.cc:2369] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:08:09.221570 13648 raft_consensus.cc:2454] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df78540f1e544a9da2dcc479af683031 in term 1.
I20260929 01:08:09.221604 13792 raft_consensus.cc:2454] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df78540f1e544a9da2dcc479af683031 in term 1.
I20260929 01:08:09.221971 13858 leader_election.cc:302] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de8e6eb4bdc8453ba8eb035d8058fa13, df78540f1e544a9da2dcc479af683031; no voters:
I20260929 01:08:09.222285 14012 raft_consensus.cc:2781] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:08:09.222520 14012 raft_consensus.cc:686] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [term 1 LEADER]: Becoming Leader. State: Replica: df78540f1e544a9da2dcc479af683031, State: Running, Role: LEADER
I20260929 01:08:09.222770 14012 consensus_queue.cc:237] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.225589 13488 catalog_manager.cc:5495] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 reported cstate change: term changed from 0 to 1, leader changed from <none> to df78540f1e544a9da2dcc479af683031 (127.5.198.195). New cstate: current_term: 1 leader_uuid: "df78540f1e544a9da2dcc479af683031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } health_report { overall_health: HEALTHY } } }
Sep 29 01:08:09 dist-test-slave-41cl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.254@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM
I20260929 01:08:09.281179 14037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:09.266858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38465 (local address 127.0.0.1:53948)
0929 01:08:09.267511 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:09.267532 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:09.267689 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:09.268046 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:09.268052 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:09.268795 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:09.268986 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:09.268994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.269994 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.270001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:09.270936 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:09.270948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.271625 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.271631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:09.271809 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:09.272478 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:09.272497 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:09.274877 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:09.278681 (+ 3804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:09.278689 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:09.278695 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:09.279113 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:09.279509 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:09.279512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:09.279514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:09.279654 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:09.280280 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:09.280287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:09.280439 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 01:08:09.280664 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 01:08:09.280865 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":334,"threads_started":1}
I20260929 01:08:09.281477 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:09.266876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.5.198.254:38465)
0929 01:08:09.267781 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:09.267789 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:09.267821 (+ 32us) server_negotiation.cc:407] Connection header received
0929 01:08:09.267858 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:09.267862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:09.267914 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:09.268020 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:09.269124 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.269859 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.271083 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.271472 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.271607 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:09.275086 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:09.275093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:09.275099 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:09.275135 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:09.278544 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:09.279234 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:09.279241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:09.279244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:09.279394 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:09.279768 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:09.279784 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:09.279787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:09.280121 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:09.280286 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:09.280797 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 01:08:09.281030 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":639,"threads_started":1}
I20260929 01:08:09.286289 13488 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:53948:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 01:08:09.287335 13488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:08:09.294147 13628 tablet_service.cc:1467] Processing CreateTablet for tablet ec9ccf4a9ed148f2a284810969acddee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bb005181e5564275a9afb6c8432118b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:08:09.294463 13906 tablet_service.cc:1467] Processing CreateTablet for tablet ec9ccf4a9ed148f2a284810969acddee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bb005181e5564275a9afb6c8432118b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:08:09.294746 13628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec9ccf4a9ed148f2a284810969acddee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:09.294987 13906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec9ccf4a9ed148f2a284810969acddee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:09.301565 14002 tablet_bootstrap.cc:492] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031: Bootstrap starting.
I20260929 01:08:09.301998 14001 tablet_bootstrap.cc:492] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21: Bootstrap starting.
I20260929 01:08:09.302022 13767 tablet_service.cc:1467] Processing CreateTablet for tablet ec9ccf4a9ed148f2a284810969acddee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bb005181e5564275a9afb6c8432118b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:08:09.302515 13767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec9ccf4a9ed148f2a284810969acddee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:09.302953 14002 tablet_bootstrap.cc:654] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:09.303556 14001 tablet_bootstrap.cc:654] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:09.304992 14002 tablet_bootstrap.cc:492] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031: No bootstrap required, opened a new log
I20260929 01:08:09.305153 14002 ts_tablet_manager.cc:1397] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 01:08:09.305912 14002 raft_consensus.cc:348] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.306171 14002 raft_consensus.cc:374] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:09.306258 14002 raft_consensus.cc:729] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df78540f1e544a9da2dcc479af683031, State: Initialized, Role: FOLLOWER
I20260929 01:08:09.306538 14002 consensus_queue.cc:260] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.307176 14002 ts_tablet_manager.cc:1428] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 01:08:09.307016 14012 tablet_replica.cc:439] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } }
I20260929 01:08:09.307328 14012 tablet_replica.cc:442] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:08:09.310392 14004 tablet_bootstrap.cc:492] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13: Bootstrap starting.
I20260929 01:08:09.311827 14004 tablet_bootstrap.cc:654] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:09.313681 14004 tablet_bootstrap.cc:492] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13: No bootstrap required, opened a new log
I20260929 01:08:09.313854 14004 ts_tablet_manager.cc:1397] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 01:08:09.314636 14004 raft_consensus.cc:348] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.315088 14004 raft_consensus.cc:374] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:09.315186 14004 raft_consensus.cc:729] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de8e6eb4bdc8453ba8eb035d8058fa13, State: Initialized, Role: FOLLOWER
I20260929 01:08:09.315434 14004 consensus_queue.cc:260] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.315549 14001 tablet_bootstrap.cc:492] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21: No bootstrap required, opened a new log
I20260929 01:08:09.315706 14001 ts_tablet_manager.cc:1397] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260929 01:08:09.315836 14009 tablet_replica.cc:439] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } }
I20260929 01:08:09.316159 14009 tablet_replica.cc:442] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:08:09.316418 14001 raft_consensus.cc:348] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.316694 14001 raft_consensus.cc:374] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:09.316787 14001 raft_consensus.cc:729] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1414d21b20f4c1aa99853821d1e0d21, State: Initialized, Role: FOLLOWER
I20260929 01:08:09.317104 14001 consensus_queue.cc:260] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.317531 14007 tablet_replica.cc:439] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } }
I20260929 01:08:09.317858 14007 tablet_replica.cc:442] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:08:09.318002 14004 ts_tablet_manager.cc:1428] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 01:08:09.318401 14001 ts_tablet_manager.cc:1428] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 01:08:09.601495 14012 raft_consensus.cc:482] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:08:09.601722 14012 raft_consensus.cc:504] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.602377 14012 leader_election.cc:288] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1414d21b20f4c1aa99853821d1e0d21 (127.5.198.193:36481), de8e6eb4bdc8453ba8eb035d8058fa13 (127.5.198.194:45535)
I20260929 01:08:09.602912 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec9ccf4a9ed148f2a284810969acddee" candidate_uuid: "df78540f1e544a9da2dcc479af683031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" is_pre_election: true
I20260929 01:08:09.602967 13648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec9ccf4a9ed148f2a284810969acddee" candidate_uuid: "df78540f1e544a9da2dcc479af683031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1414d21b20f4c1aa99853821d1e0d21" is_pre_election: true
I20260929 01:08:09.603163 13792 raft_consensus.cc:2454] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df78540f1e544a9da2dcc479af683031 in term 0.
I20260929 01:08:09.603296 13648 raft_consensus.cc:2454] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df78540f1e544a9da2dcc479af683031 in term 0.
I20260929 01:08:09.603605 13858 leader_election.cc:302] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de8e6eb4bdc8453ba8eb035d8058fa13, df78540f1e544a9da2dcc479af683031; no voters:
I20260929 01:08:09.603909 14012 raft_consensus.cc:2781] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:08:09.604031 14012 raft_consensus.cc:482] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:08:09.604148 14012 raft_consensus.cc:3037] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:09.606084 14012 raft_consensus.cc:504] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.606628 14012 leader_election.cc:288] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [CANDIDATE]: Term 1 election: Requested vote from peers f1414d21b20f4c1aa99853821d1e0d21 (127.5.198.193:36481), de8e6eb4bdc8453ba8eb035d8058fa13 (127.5.198.194:45535)
I20260929 01:08:09.607039 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec9ccf4a9ed148f2a284810969acddee" candidate_uuid: "df78540f1e544a9da2dcc479af683031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13"
I20260929 01:08:09.607043 13648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec9ccf4a9ed148f2a284810969acddee" candidate_uuid: "df78540f1e544a9da2dcc479af683031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1414d21b20f4c1aa99853821d1e0d21"
I20260929 01:08:09.607241 13648 raft_consensus.cc:3037] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:09.607229 13792 raft_consensus.cc:3037] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:09.609056 13792 raft_consensus.cc:2454] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df78540f1e544a9da2dcc479af683031 in term 1.
I20260929 01:08:09.609179 13648 raft_consensus.cc:2454] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df78540f1e544a9da2dcc479af683031 in term 1.
I20260929 01:08:09.609450 13858 leader_election.cc:302] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de8e6eb4bdc8453ba8eb035d8058fa13, df78540f1e544a9da2dcc479af683031; no voters:
I20260929 01:08:09.609750 14012 raft_consensus.cc:2781] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:08:09.609908 14012 raft_consensus.cc:686] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [term 1 LEADER]: Becoming Leader. State: Replica: df78540f1e544a9da2dcc479af683031, State: Running, Role: LEADER
I20260929 01:08:09.610172 14012 consensus_queue.cc:237] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.611656 14021 tablet_replica.cc:439] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031: TxnStatusTablet state changed. Reason: New leader df78540f1e544a9da2dcc479af683031. Latest consensus state: current_term: 1 leader_uuid: "df78540f1e544a9da2dcc479af683031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } }
I20260929 01:08:09.611904 14021 tablet_replica.cc:442] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:08:09.612434 14043 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 01:08:09.612679 14043 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 01:08:09.613168 13488 catalog_manager.cc:5495] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 reported cstate change: term changed from 0 to 1, leader changed from <none> to df78540f1e544a9da2dcc479af683031 (127.5.198.195). New cstate: current_term: 1 leader_uuid: "df78540f1e544a9da2dcc479af683031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } health_report { overall_health: HEALTHY } } }
I20260929 01:08:09.676595 14037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:09.671303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:44017 (local address 127.0.0.1:46706)
0929 01:08:09.671524 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:09.671547 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:09.671705 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:09.672188 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:09.672192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:09.672207 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:08:09.672410 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:09.672428 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.673594 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.673602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:09.674541 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:09.674552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.675346 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.675352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:09.675526 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:09.675578 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:08:09.675992 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:08:09.676013 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:08:09.676111 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 01:08:09.676249 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 01:08:09.676656 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:09.671306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.5.198.195:44017)
0929 01:08:09.671904 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:09.671909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:09.671925 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:08:09.671989 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:09.671992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:09.672052 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:09.672163 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:08:09.672601 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.673461 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.674686 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.675175 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.675329 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:09.675771 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:08:09.675869 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:08:09.675990 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:09.676130 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 01:08:09.676203 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":320,"threads_started":1}
I20260929 01:08:09.680531 13648 raft_consensus.cc:1261] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Refusing update from remote peer df78540f1e544a9da2dcc479af683031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:08:09.680626 13792 raft_consensus.cc:1261] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 FOLLOWER]: Refusing update from remote peer df78540f1e544a9da2dcc479af683031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:08:09.681192 14021 consensus_queue.cc:1035] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:08:09.681528 14012 consensus_queue.cc:1035] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:08:09.689319 14009 tablet_replica.cc:439] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13: TxnStatusTablet state changed. Reason: New leader df78540f1e544a9da2dcc479af683031. Latest consensus state: current_term: 1 leader_uuid: "df78540f1e544a9da2dcc479af683031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } }
I20260929 01:08:09.689589 14009 tablet_replica.cc:442] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:08:09.692584 14048 mvcc.cc:205] Tried to move back new op lower bound from 7334478191325609984 to 7334478191045287936. Current Snapshot: MvccSnapshot[applied={T|T < 7334478191325609984}]
I20260929 01:08:09.700198 14009 tablet_replica.cc:439] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df78540f1e544a9da2dcc479af683031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } }
I20260929 01:08:09.700552 14009 tablet_replica.cc:442] T ec9ccf4a9ed148f2a284810969acddee P de8e6eb4bdc8453ba8eb035d8058fa13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:08:09.704070 14007 tablet_replica.cc:439] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21: TxnStatusTablet state changed. Reason: New leader df78540f1e544a9da2dcc479af683031. Latest consensus state: current_term: 1 leader_uuid: "df78540f1e544a9da2dcc479af683031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } }
I20260929 01:08:09.704321 14007 tablet_replica.cc:442] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:08:09.706508 14021 tablet_replica.cc:439] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df78540f1e544a9da2dcc479af683031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } }
I20260929 01:08:09.706779 14021 tablet_replica.cc:442] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:08:09.711200 14021 tablet_replica.cc:439] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df78540f1e544a9da2dcc479af683031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } }
I20260929 01:08:09.711534 14021 tablet_replica.cc:442] T ec9ccf4a9ed148f2a284810969acddee P df78540f1e544a9da2dcc479af683031: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:08:09.715411 14007 tablet_replica.cc:439] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df78540f1e544a9da2dcc479af683031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } } }
I20260929 01:08:09.715663 14007 tablet_replica.cc:442] T ec9ccf4a9ed148f2a284810969acddee P f1414d21b20f4c1aa99853821d1e0d21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:08:09.720077 14012 consensus_queue.cc:1035] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:08:09.733150 14021 consensus_queue.cc:1035] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:08:09.742283 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:09.731943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:41889 (local address 127.5.198.195:44017)
0929 01:08:09.732358 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:09.732364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:09.736094 (+ 3730us) server_negotiation.cc:407] Connection header received
0929 01:08:09.736532 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:09.736537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:09.736606 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:09.736664 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:08:09.737418 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.738487 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.740581 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.740987 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.741076 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:09.741593 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:08:09.741677 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:08:09.741737 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:09.741921 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 01:08:09.741988 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295}
I20260929 01:08:09.742494 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:09.731504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:44017 (local address 127.5.198.195:41889)
0929 01:08:09.736008 (+ 4504us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:09.736029 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:09.736353 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:09.736852 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:09.736857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:09.736868 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:08:09.737256 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:09.737268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.738606 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.738612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:09.740426 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:09.740440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.741134 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.741140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:09.741345 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:09.741424 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:08:09.741771 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:08:09.741781 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 01:08:09.741882 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 01:08:09.742016 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3763,"thread_start_us":3327,"threads_started":1}
I20260929 01:08:09.802055 13489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:08:09.802313 13489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:08:09.805747 13489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1: Sending DeleteTablet for 3 replicas of tablet bbaf5b62ef9c46ce8e938421f6ee0fc1
I20260929 01:08:09.806681 13767 tablet_service.cc:1514] Processing DeleteTablet for tablet bbaf5b62ef9c46ce8e938421f6ee0fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:09 UTC) from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:52984
I20260929 01:08:09.807035 13905 tablet_service.cc:1514] Processing DeleteTablet for tablet bbaf5b62ef9c46ce8e938421f6ee0fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:09 UTC) from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:46694
I20260929 01:08:09.807017 13628 tablet_service.cc:1514] Processing DeleteTablet for tablet bbaf5b62ef9c46ce8e938421f6ee0fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:09 UTC) from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:37234
I20260929 01:08:09.807363 14063 tablet_replica.cc:330] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13: stopping tablet replica
I20260929 01:08:09.807574 14063 raft_consensus.cc:2229] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:08:09.807811 14063 raft_consensus.cc:2258] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:09.817628 14063 ts_tablet_manager.cc:1905] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:09.828966 14065 tablet_replica.cc:330] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031: stopping tablet replica
I20260929 01:08:09.829244 14065 raft_consensus.cc:2229] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:08:09.829664 14065 raft_consensus.cc:2258] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:09.829705 14063 ts_tablet_manager.cc:1918] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:08:09.830214 14064 tablet_replica.cc:330] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21: stopping tablet replica
I20260929 01:08:09.830470 14064 raft_consensus.cc:2229] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:08:09.830734 14064 raft_consensus.cc:2258] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:09.830808 14065 ts_tablet_manager.cc:1905] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:09.829865 14063 log.cc:1196] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/bbaf5b62ef9c46ce8e938421f6ee0fc1
I20260929 01:08:09.831765 14064 ts_tablet_manager.cc:1905] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:09.831904 14063 ts_tablet_manager.cc:1939] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P de8e6eb4bdc8453ba8eb035d8058fa13: Deleting consensus metadata
I20260929 01:08:09.833025 13475 catalog_manager.cc:4841] TS de8e6eb4bdc8453ba8eb035d8058fa13 (127.5.198.194:45535): tablet bbaf5b62ef9c46ce8e938421f6ee0fc1 (table test-table [id=408e61a68e3a4fbf8eeaf64c4a04aad4]) successfully deleted
I20260929 01:08:09.838507 14064 ts_tablet_manager.cc:1918] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:08:09.838809 14064 log.cc:1196] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/bbaf5b62ef9c46ce8e938421f6ee0fc1
I20260929 01:08:09.839286 14064 ts_tablet_manager.cc:1939] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P f1414d21b20f4c1aa99853821d1e0d21: Deleting consensus metadata
I20260929 01:08:09.840407 14065 ts_tablet_manager.cc:1918] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:08:09.840476 13474 catalog_manager.cc:4841] TS f1414d21b20f4c1aa99853821d1e0d21 (127.5.198.193:36481): tablet bbaf5b62ef9c46ce8e938421f6ee0fc1 (table test-table [id=408e61a68e3a4fbf8eeaf64c4a04aad4]) successfully deleted
I20260929 01:08:09.840583 14065 log.cc:1196] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/bbaf5b62ef9c46ce8e938421f6ee0fc1
I20260929 01:08:09.841027 14065 ts_tablet_manager.cc:1939] T bbaf5b62ef9c46ce8e938421f6ee0fc1 P df78540f1e544a9da2dcc479af683031: Deleting consensus metadata
I20260929 01:08:09.842065 13475 catalog_manager.cc:4841] TS df78540f1e544a9da2dcc479af683031 (127.5.198.195:44017): tablet bbaf5b62ef9c46ce8e938421f6ee0fc1 (table test-table [id=408e61a68e3a4fbf8eeaf64c4a04aad4]) successfully deleted
Sep 29 01:08:09 dist-test-slave-41cl krb5kdc[13445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 01:08:09 dist-test-slave-41cl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:09 dist-test-slave-41cl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 01:08:09.887405 14073 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:09.875306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53956 (local address 127.5.198.254:38465)
0929 01:08:09.876080 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:09.876084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:09.876101 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:08:09.876154 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:09.876158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:09.876227 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:09.876315 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:09.877994 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.878751 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.880661 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.881068 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.881163 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:09.887076 (+ 5913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53956: BlockingRecv error: recv got EOF from 127.0.0.1:53956 (error 108)
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":571,"threads_started":1}
Sep 29 01:08:09 dist-test-slave-41cl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM
I20260929 01:08:09.905646 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:09.890032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.5.198.254:38465)
0929 01:08:09.890219 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:09.890224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:09.890474 (+ 250us) server_negotiation.cc:407] Connection header received
0929 01:08:09.890634 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:09.890638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:09.890696 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:09.890792 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:09.892218 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.892992 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.894719 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.895105 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.895193 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:09.899524 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:09.899533 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:09.899539 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:09.899580 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:09.903104 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:09.903706 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:09.903713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:09.903717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:09.903879 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:09.904218 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:09.904223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:09.904227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:09.904574 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:09.904838 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:09.905081 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 01:08:09.905300 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:08:09.909391 13489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:08:09.910112 13489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:08:09.917065 13628 tablet_service.cc:1467] Processing CreateTablet for tablet 3466147820db4f01a8234167f8c5316a (DEFAULT_TABLE table=test-table [id=1034ca61dad54c69a2f5a6f5245275c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:09.917181 13767 tablet_service.cc:1467] Processing CreateTablet for tablet 3466147820db4f01a8234167f8c5316a (DEFAULT_TABLE table=test-table [id=1034ca61dad54c69a2f5a6f5245275c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:09.917335 13905 tablet_service.cc:1467] Processing CreateTablet for tablet 3466147820db4f01a8234167f8c5316a (DEFAULT_TABLE table=test-table [id=1034ca61dad54c69a2f5a6f5245275c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:09.917528 13767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3466147820db4f01a8234167f8c5316a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:09.917528 13628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3466147820db4f01a8234167f8c5316a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:09.917762 13905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3466147820db4f01a8234167f8c5316a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:09.929558 14079 tablet_bootstrap.cc:492] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13: Bootstrap starting.
I20260929 01:08:09.930879 14080 tablet_bootstrap.cc:492] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031: Bootstrap starting.
I20260929 01:08:09.931260 14079 tablet_bootstrap.cc:654] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:09.934135 14079 tablet_bootstrap.cc:492] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13: No bootstrap required, opened a new log
I20260929 01:08:09.934290 14079 ts_tablet_manager.cc:1397] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 01:08:09.934844 14080 tablet_bootstrap.cc:654] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:09.935076 14079 raft_consensus.cc:348] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.935324 14079 raft_consensus.cc:374] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:09.935424 14079 raft_consensus.cc:729] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de8e6eb4bdc8453ba8eb035d8058fa13, State: Initialized, Role: FOLLOWER
I20260929 01:08:09.935684 14079 consensus_queue.cc:260] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.936416 14081 tablet_bootstrap.cc:492] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21: Bootstrap starting.
I20260929 01:08:09.938627 14079 ts_tablet_manager.cc:1428] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 01:08:09.938715 14081 tablet_bootstrap.cc:654] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:09.941243 14080 tablet_bootstrap.cc:492] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031: No bootstrap required, opened a new log
I20260929 01:08:09.941404 14080 ts_tablet_manager.cc:1397] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260929 01:08:09.942279 14080 raft_consensus.cc:348] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.942566 14080 raft_consensus.cc:374] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:09.942680 14080 raft_consensus.cc:729] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df78540f1e544a9da2dcc479af683031, State: Initialized, Role: FOLLOWER
I20260929 01:08:09.942983 14080 consensus_queue.cc:260] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.943573 14080 ts_tablet_manager.cc:1428] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:08:09.945243 14081 tablet_bootstrap.cc:492] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21: No bootstrap required, opened a new log
I20260929 01:08:09.945382 14081 ts_tablet_manager.cc:1397] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20260929 01:08:09.946110 14081 raft_consensus.cc:348] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.946385 14081 raft_consensus.cc:374] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:09.946462 14081 raft_consensus.cc:729] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1414d21b20f4c1aa99853821d1e0d21, State: Initialized, Role: FOLLOWER
I20260929 01:08:09.946668 14081 consensus_queue.cc:260] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.947149 14081 ts_tablet_manager.cc:1428] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 01:08:09.948971 14009 raft_consensus.cc:482] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:08:09.948988 14007 raft_consensus.cc:482] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:08:09.949147 14007 raft_consensus.cc:504] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.949157 14009 raft_consensus.cc:504] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.950729 14007 leader_election.cc:288] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de8e6eb4bdc8453ba8eb035d8058fa13 (127.5.198.194:45535), df78540f1e544a9da2dcc479af683031 (127.5.198.195:44017)
I20260929 01:08:09.950794 14009 leader_election.cc:288] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1414d21b20f4c1aa99853821d1e0d21 (127.5.198.193:36481), df78540f1e544a9da2dcc479af683031 (127.5.198.195:44017)
I20260929 01:08:09.989225 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:09.953596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:34453 (local address 127.5.198.195:44017)
0929 01:08:09.953800 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:09.953810 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:09.978537 (+ 24727us) server_negotiation.cc:407] Connection header received
0929 01:08:09.979094 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:09.979103 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:09.979194 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:09.979254 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:09.980072 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.981465 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.985684 (+ 4219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.986988 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.987123 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:09.987196 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:09.988617 (+ 1421us) server_negotiation.cc:299] Negotiation successful
0929 01:08:09.988807 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:08:09.990583 14086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:09.951191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:44017 (local address 127.5.198.194:34453)
0929 01:08:09.978404 (+ 27213us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:09.978432 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:09.978866 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:09.979454 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:09.979461 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:09.979493 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:09.979885 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:09.979897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.981711 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.981720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:09.984415 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:09.984428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:09.987541 (+ 3113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:09.987550 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:09.987800 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:09.987852 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 01:08:09.989533 (+ 1681us) client_negotiation.cc:240] Negotiation successful
0929 01:08:09.989722 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27023,"thread_start_us":27384,"threads_started":1}
I20260929 01:08:09.991485 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3466147820db4f01a8234167f8c5316a" candidate_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df78540f1e544a9da2dcc479af683031" is_pre_election: true
I20260929 01:08:09.992012 13931 raft_consensus.cc:2454] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de8e6eb4bdc8453ba8eb035d8058fa13 in term 0.
I20260929 01:08:09.992949 13719 leader_election.cc:302] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de8e6eb4bdc8453ba8eb035d8058fa13, df78540f1e544a9da2dcc479af683031; no voters:
I20260929 01:08:09.993381 14009 raft_consensus.cc:2781] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:08:09.993513 14009 raft_consensus.cc:482] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:08:09.993607 14009 raft_consensus.cc:3037] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:09.997298 14009 raft_consensus.cc:504] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:09.998090 14009 leader_election.cc:288] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [CANDIDATE]: Term 1 election: Requested vote from peers f1414d21b20f4c1aa99853821d1e0d21 (127.5.198.193:36481), df78540f1e544a9da2dcc479af683031 (127.5.198.195:44017)
I20260929 01:08:09.998536 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3466147820db4f01a8234167f8c5316a" candidate_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df78540f1e544a9da2dcc479af683031"
I20260929 01:08:09.998756 13931 raft_consensus.cc:3037] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:10.002010 13931 raft_consensus.cc:2454] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de8e6eb4bdc8453ba8eb035d8058fa13 in term 1.
I20260929 01:08:10.002449 13719 leader_election.cc:302] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de8e6eb4bdc8453ba8eb035d8058fa13, df78540f1e544a9da2dcc479af683031; no voters:
I20260929 01:08:10.003131 14009 raft_consensus.cc:2781] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:08:10.007421 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:09.950899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:45535 (local address 127.5.198.193:57281)
0929 01:08:09.996222 (+ 45323us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:09.996240 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:09.996344 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:09.997335 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:09.997340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:09.997365 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:09.997691 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:09.997706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.002061 (+ 4355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.002066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:10.004530 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:10.004551 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.006587 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.006594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:10.006765 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:10.006797 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 01:08:10.006891 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 01:08:10.007034 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45207,"thread_start_us":51161,"threads_started":1}
I20260929 01:08:10.011644 14086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:09.951356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:36481 (local address 127.5.198.194:51209)
0929 01:08:09.991835 (+ 40479us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:09.991857 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:09.992044 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:09.997434 (+ 5390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:09.997438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:09.997465 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:09.997632 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:09.997645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.007049 (+ 9404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.007054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:10.008692 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:10.008714 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.010911 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.010917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:10.011113 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:10.011145 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 01:08:10.011212 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 01:08:10.011343 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40343,"thread_start_us":45693,"threads_started":1}
I20260929 01:08:10.013887 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:09.951659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:51209 (local address 127.5.198.193:36481)
0929 01:08:09.993686 (+ 42027us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:09.993692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:09.993717 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:08:09.993783 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:09.993790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:09.993871 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:09.993974 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:09.997813 (+ 3839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.006894 (+ 9081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.009287 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.010586 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.010748 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:10.010814 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:10.013417 (+ 2603us) server_negotiation.cc:299] Negotiation successful
0929 01:08:10.013553 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41876,"thread_start_us":45643,"threads_started":1}
I20260929 01:08:10.014369 13648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3466147820db4f01a8234167f8c5316a" candidate_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1414d21b20f4c1aa99853821d1e0d21" is_pre_election: true
I20260929 01:08:10.014605 13648 raft_consensus.cc:2454] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de8e6eb4bdc8453ba8eb035d8058fa13 in term 0.
I20260929 01:08:10.015120 13648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3466147820db4f01a8234167f8c5316a" candidate_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1414d21b20f4c1aa99853821d1e0d21"
I20260929 01:08:10.015322 13648 raft_consensus.cc:3037] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:10.016371 14009 raft_consensus.cc:686] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 LEADER]: Becoming Leader. State: Replica: de8e6eb4bdc8453ba8eb035d8058fa13, State: Running, Role: LEADER
I20260929 01:08:10.016916 14009 consensus_queue.cc:237] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:10.018326 13648 raft_consensus.cc:2454] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de8e6eb4bdc8453ba8eb035d8058fa13 in term 1.
I20260929 01:08:10.020749 14087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:09.951270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:57281 (local address 127.5.198.194:45535)
0929 01:08:09.995801 (+ 44531us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:09.995807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:09.996454 (+ 647us) server_negotiation.cc:407] Connection header received
0929 01:08:09.996546 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:09.996551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:09.996611 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:09.996687 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:10.000613 (+ 3926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.001932 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.004756 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.006121 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.006350 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:10.006459 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:10.020213 (+ 13754us) server_negotiation.cc:299] Negotiation successful
0929 01:08:10.020363 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44394,"thread_start_us":44538,"threads_started":1}
I20260929 01:08:10.021239 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3466147820db4f01a8234167f8c5316a" candidate_uuid: "f1414d21b20f4c1aa99853821d1e0d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" is_pre_election: true
I20260929 01:08:10.022550 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:10.002651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:44017 (local address 127.5.198.193:51525)
0929 01:08:10.007648 (+ 4997us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:10.007666 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:10.009290 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:10.016963 (+ 7673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:10.016968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:10.016996 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:10.017141 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:10.017150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.018623 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.018628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:10.020005 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:10.020024 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.021849 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.021854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:10.022031 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:10.022062 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 01:08:10.022147 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 01:08:10.022275 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4903,"thread_start_us":9967,"threads_started":1}
I20260929 01:08:10.023123 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:10.002822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:51525 (local address 127.5.198.195:44017)
0929 01:08:10.002973 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:10.002979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:10.007734 (+ 4755us) server_negotiation.cc:407] Connection header received
0929 01:08:10.016607 (+ 8873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:10.016613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:10.016697 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:10.016763 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:10.017294 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.018420 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.020157 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.021554 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.021690 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:10.021757 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:10.022739 (+ 982us) server_negotiation.cc:299] Negotiation successful
0929 01:08:10.022878 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:08:10.023535 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3466147820db4f01a8234167f8c5316a" candidate_uuid: "f1414d21b20f4c1aa99853821d1e0d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df78540f1e544a9da2dcc479af683031" is_pre_election: true
I20260929 01:08:10.023837 13931 raft_consensus.cc:2379] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1414d21b20f4c1aa99853821d1e0d21 in current term 1: Already voted for candidate de8e6eb4bdc8453ba8eb035d8058fa13 in this term.
I20260929 01:08:10.024382 13579 leader_election.cc:302] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1414d21b20f4c1aa99853821d1e0d21; no voters: de8e6eb4bdc8453ba8eb035d8058fa13, df78540f1e544a9da2dcc479af683031
I20260929 01:08:10.024823 14007 raft_consensus.cc:2726] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 01:08:10.030895 13488 catalog_manager.cc:5495] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 reported cstate change: term changed from 0 to 1, leader changed from <none> to de8e6eb4bdc8453ba8eb035d8058fa13 (127.5.198.194). New cstate: current_term: 1 leader_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } health_report { overall_health: UNKNOWN } } }
I20260929 01:08:10.064735 14087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:10.058498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.5.198.194:45535)
0929 01:08:10.058686 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:10.058692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:10.058810 (+ 118us) server_negotiation.cc:407] Connection header received
0929 01:08:10.059000 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:10.059005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:10.059102 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:10.059274 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:08:10.059636 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.060736 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.062129 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.062722 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.062882 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:10.063423 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:08:10.064155 (+ 732us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:08:10.064296 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:10.064365 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 01:08:10.064447 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:08:10.068393 13648 raft_consensus.cc:1261] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Refusing update from remote peer de8e6eb4bdc8453ba8eb035d8058fa13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:08:10.068701 13931 raft_consensus.cc:1261] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031 [term 1 FOLLOWER]: Refusing update from remote peer de8e6eb4bdc8453ba8eb035d8058fa13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:08:10.069155 14009 consensus_queue.cc:1035] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:08:10.069489 14091 consensus_queue.cc:1035] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:08:10.081544 14047 mvcc.cc:205] Tried to move back new op lower bound from 7334478192913498112 to 7334478192714579968. Current Snapshot: MvccSnapshot[applied={T|T < 7334478192913498112}]
I20260929 01:08:10.098387 13488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:08:10.098634 13488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:08:10.101657 13488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1: Sending DeleteTablet for 3 replicas of tablet 3466147820db4f01a8234167f8c5316a
I20260929 01:08:10.102489 13767 tablet_service.cc:1514] Processing DeleteTablet for tablet 3466147820db4f01a8234167f8c5316a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:10 UTC) from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:52984
I20260929 01:08:10.102844 13628 tablet_service.cc:1514] Processing DeleteTablet for tablet 3466147820db4f01a8234167f8c5316a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:10 UTC) from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:37234
I20260929 01:08:10.102965 13905 tablet_service.cc:1514] Processing DeleteTablet for tablet 3466147820db4f01a8234167f8c5316a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:10 UTC) from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:46694
I20260929 01:08:10.103046 14063 tablet_replica.cc:330] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13: stopping tablet replica
I20260929 01:08:10.103163 14064 tablet_replica.cc:330] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21: stopping tablet replica
I20260929 01:08:10.103250 14065 tablet_replica.cc:330] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031: stopping tablet replica
I20260929 01:08:10.103255 14063 raft_consensus.cc:2229] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:08:10.103350 14064 raft_consensus.cc:2229] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:08:10.103448 14065 raft_consensus.cc:2229] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:08:10.103569 14064 raft_consensus.cc:2258] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:10.103664 14065 raft_consensus.cc:2258] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:10.103783 14063 raft_consensus.cc:2258] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:10.104758 14064 ts_tablet_manager.cc:1905] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:10.105116 14065 ts_tablet_manager.cc:1905] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:10.105691 13488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:08:10.106807 13488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:08:10.109048 14063 ts_tablet_manager.cc:1905] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:10.111527 14065 ts_tablet_manager.cc:1918] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:08:10.111660 14065 log.cc:1196] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/3466147820db4f01a8234167f8c5316a
I20260929 01:08:10.112039 14065 ts_tablet_manager.cc:1939] T 3466147820db4f01a8234167f8c5316a P df78540f1e544a9da2dcc479af683031: Deleting consensus metadata
I20260929 01:08:10.113122 13475 catalog_manager.cc:4841] TS df78540f1e544a9da2dcc479af683031 (127.5.198.195:44017): tablet 3466147820db4f01a8234167f8c5316a (table test-table [id=1034ca61dad54c69a2f5a6f5245275c0]) successfully deleted
I20260929 01:08:10.113679 14064 ts_tablet_manager.cc:1918] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:08:10.113811 14064 log.cc:1196] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/3466147820db4f01a8234167f8c5316a
I20260929 01:08:10.114202 14064 ts_tablet_manager.cc:1939] T 3466147820db4f01a8234167f8c5316a P f1414d21b20f4c1aa99853821d1e0d21: Deleting consensus metadata
I20260929 01:08:10.115381 14063 ts_tablet_manager.cc:1918] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:08:10.115515 14063 log.cc:1196] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/3466147820db4f01a8234167f8c5316a
I20260929 01:08:10.115564 13474 catalog_manager.cc:4841] TS f1414d21b20f4c1aa99853821d1e0d21 (127.5.198.193:36481): tablet 3466147820db4f01a8234167f8c5316a (table test-table [id=1034ca61dad54c69a2f5a6f5245275c0]) successfully deleted
I20260929 01:08:10.115892 14063 ts_tablet_manager.cc:1939] T 3466147820db4f01a8234167f8c5316a P de8e6eb4bdc8453ba8eb035d8058fa13: Deleting consensus metadata
I20260929 01:08:10.117766 13475 catalog_manager.cc:4841] TS de8e6eb4bdc8453ba8eb035d8058fa13 (127.5.198.194:45535): tablet 3466147820db4f01a8234167f8c5316a (table test-table [id=1034ca61dad54c69a2f5a6f5245275c0]) successfully deleted
I20260929 01:08:10.118932 13628 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5e7a533af24c7b8c6469f096b30807 (DEFAULT_TABLE table=test-table [id=5234f92c0b5545f38273dc13ddc7a654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:10.118994 13767 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5e7a533af24c7b8c6469f096b30807 (DEFAULT_TABLE table=test-table [id=5234f92c0b5545f38273dc13ddc7a654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:10.119379 13628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5e7a533af24c7b8c6469f096b30807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:10.119474 13767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5e7a533af24c7b8c6469f096b30807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:10.121529 13905 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5e7a533af24c7b8c6469f096b30807 (DEFAULT_TABLE table=test-table [id=5234f92c0b5545f38273dc13ddc7a654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:10.121958 13905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5e7a533af24c7b8c6469f096b30807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:10.125695 14079 tablet_bootstrap.cc:492] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13: Bootstrap starting.
I20260929 01:08:10.126561 14081 tablet_bootstrap.cc:492] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21: Bootstrap starting.
I20260929 01:08:10.127178 14079 tablet_bootstrap.cc:654] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:10.128176 14081 tablet_bootstrap.cc:654] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:10.130139 14081 tablet_bootstrap.cc:492] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21: No bootstrap required, opened a new log
I20260929 01:08:10.130293 14081 ts_tablet_manager.cc:1397] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 01:08:10.130988 14081 raft_consensus.cc:348] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:10.131266 14081 raft_consensus.cc:374] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:10.131379 14081 raft_consensus.cc:729] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1414d21b20f4c1aa99853821d1e0d21, State: Initialized, Role: FOLLOWER
I20260929 01:08:10.131660 14081 consensus_queue.cc:260] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:10.132323 14081 ts_tablet_manager.cc:1428] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 01:08:10.132452 14080 tablet_bootstrap.cc:492] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031: Bootstrap starting.
I20260929 01:08:10.134155 14080 tablet_bootstrap.cc:654] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:10.137207 14080 tablet_bootstrap.cc:492] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031: No bootstrap required, opened a new log
I20260929 01:08:10.137370 14080 ts_tablet_manager.cc:1397] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 01:08:10.138024 14080 raft_consensus.cc:348] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:10.138201 14080 raft_consensus.cc:374] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:10.138303 14080 raft_consensus.cc:729] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df78540f1e544a9da2dcc479af683031, State: Initialized, Role: FOLLOWER
I20260929 01:08:10.138525 14080 consensus_queue.cc:260] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:10.139042 14080 ts_tablet_manager.cc:1428] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:08:10.143991 14079 tablet_bootstrap.cc:492] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13: No bootstrap required, opened a new log
I20260929 01:08:10.144174 14079 ts_tablet_manager.cc:1397] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.007s
I20260929 01:08:10.144901 14079 raft_consensus.cc:348] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:10.145125 14079 raft_consensus.cc:374] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:10.145206 14079 raft_consensus.cc:729] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de8e6eb4bdc8453ba8eb035d8058fa13, State: Initialized, Role: FOLLOWER
I20260929 01:08:10.145503 14079 consensus_queue.cc:260] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:10.146077 14079 ts_tablet_manager.cc:1428] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:08:10.399101 14060 raft_consensus.cc:482] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:08:10.399333 14060 raft_consensus.cc:504] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:10.400017 14060 leader_election.cc:288] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1414d21b20f4c1aa99853821d1e0d21 (127.5.198.193:36481), de8e6eb4bdc8453ba8eb035d8058fa13 (127.5.198.194:45535)
I20260929 01:08:10.400599 13648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5e7a533af24c7b8c6469f096b30807" candidate_uuid: "df78540f1e544a9da2dcc479af683031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1414d21b20f4c1aa99853821d1e0d21" is_pre_election: true
I20260929 01:08:10.400660 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5e7a533af24c7b8c6469f096b30807" candidate_uuid: "df78540f1e544a9da2dcc479af683031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" is_pre_election: true
I20260929 01:08:10.400867 13648 raft_consensus.cc:2454] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df78540f1e544a9da2dcc479af683031 in term 0.
I20260929 01:08:10.400934 13792 raft_consensus.cc:2454] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df78540f1e544a9da2dcc479af683031 in term 0.
I20260929 01:08:10.401285 13857 leader_election.cc:302] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df78540f1e544a9da2dcc479af683031, f1414d21b20f4c1aa99853821d1e0d21; no voters:
I20260929 01:08:10.401566 14060 raft_consensus.cc:2781] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:08:10.401700 14060 raft_consensus.cc:482] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:08:10.401799 14060 raft_consensus.cc:3037] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:10.404735 14060 raft_consensus.cc:504] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:10.405364 14060 leader_election.cc:288] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [CANDIDATE]: Term 1 election: Requested vote from peers f1414d21b20f4c1aa99853821d1e0d21 (127.5.198.193:36481), de8e6eb4bdc8453ba8eb035d8058fa13 (127.5.198.194:45535)
I20260929 01:08:10.405813 13648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5e7a533af24c7b8c6469f096b30807" candidate_uuid: "df78540f1e544a9da2dcc479af683031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1414d21b20f4c1aa99853821d1e0d21"
I20260929 01:08:10.406078 13648 raft_consensus.cc:3037] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:10.406004 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5e7a533af24c7b8c6469f096b30807" candidate_uuid: "df78540f1e544a9da2dcc479af683031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13"
I20260929 01:08:10.406286 13792 raft_consensus.cc:3037] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:10.408427 13792 raft_consensus.cc:2454] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df78540f1e544a9da2dcc479af683031 in term 1.
I20260929 01:08:10.408882 13858 leader_election.cc:302] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de8e6eb4bdc8453ba8eb035d8058fa13, df78540f1e544a9da2dcc479af683031; no voters:
I20260929 01:08:10.409068 13648 raft_consensus.cc:2454] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df78540f1e544a9da2dcc479af683031 in term 1.
I20260929 01:08:10.409149 14060 raft_consensus.cc:2781] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:08:10.416728 14060 raft_consensus.cc:686] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [term 1 LEADER]: Becoming Leader. State: Replica: df78540f1e544a9da2dcc479af683031, State: Running, Role: LEADER
I20260929 01:08:10.417034 14060 consensus_queue.cc:237] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } }
I20260929 01:08:10.419665 13488 catalog_manager.cc:5495] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 reported cstate change: term changed from 0 to 1, leader changed from <none> to df78540f1e544a9da2dcc479af683031 (127.5.198.195). New cstate: current_term: 1 leader_uuid: "df78540f1e544a9da2dcc479af683031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df78540f1e544a9da2dcc479af683031" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 44017 } health_report { overall_health: HEALTHY } } }
I20260929 01:08:10.444777 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:10.439553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46712 (local address 127.5.198.195:44017)
0929 01:08:10.439698 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:10.439704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:10.439812 (+ 108us) server_negotiation.cc:407] Connection header received
0929 01:08:10.440014 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:10.440019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:10.440096 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:10.440188 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:08:10.440621 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.441339 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.442938 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.443399 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.443566 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:10.443999 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:08:10.444113 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:08:10.444224 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:10.444391 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 01:08:10.444497 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:08:10.455509 13647 raft_consensus.cc:1261] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Refusing update from remote peer df78540f1e544a9da2dcc479af683031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:08:10.455554 13792 raft_consensus.cc:1261] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 FOLLOWER]: Refusing update from remote peer df78540f1e544a9da2dcc479af683031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:08:10.456038 14100 consensus_queue.cc:1035] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1414d21b20f4c1aa99853821d1e0d21" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:08:10.456406 14060 consensus_queue.cc:1035] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de8e6eb4bdc8453ba8eb035d8058fa13" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 45535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:08:10.536928 13490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:08:10.537140 13490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:08:10.540208 13490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1e376999cce41c2831666404ecdbef1: Sending DeleteTablet for 3 replicas of tablet 6b5e7a533af24c7b8c6469f096b30807
I20260929 01:08:10.541033 13767 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b5e7a533af24c7b8c6469f096b30807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:10 UTC) from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:52984
I20260929 01:08:10.541144 13628 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b5e7a533af24c7b8c6469f096b30807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:10 UTC) from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:37234
I20260929 01:08:10.541328 14063 tablet_replica.cc:330] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13: stopping tablet replica
I20260929 01:08:10.541417 14064 tablet_replica.cc:330] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21: stopping tablet replica
I20260929 01:08:10.541517 14063 raft_consensus.cc:2229] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:08:10.541589 14064 raft_consensus.cc:2229] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:08:10.541725 14063 raft_consensus.cc:2258] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:10.541771 14064 raft_consensus.cc:2258] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:10.542654 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 13562
I20260929 01:08:10.543212 14063 ts_tablet_manager.cc:1905] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:10.544337 14064 ts_tablet_manager.cc:1905] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:10.548391 13905 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b5e7a533af24c7b8c6469f096b30807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:10 UTC) from {username='oryx', principal='oryx/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:46694
I20260929 01:08:10.548691 14065 tablet_replica.cc:330] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031: stopping tablet replica
I20260929 01:08:10.548846 14065 raft_consensus.cc:2229] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:08:10.549206 14065 raft_consensus.cc:2258] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:10.550474 14065 ts_tablet_manager.cc:1905] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:10.558068 14063 ts_tablet_manager.cc:1918] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:08:10.558248 14063 log.cc:1196] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/6b5e7a533af24c7b8c6469f096b30807
I20260929 01:08:10.558787 14063 ts_tablet_manager.cc:1939] T 6b5e7a533af24c7b8c6469f096b30807 P de8e6eb4bdc8453ba8eb035d8058fa13: Deleting consensus metadata
I20260929 01:08:10.559778 14064 ts_tablet_manager.cc:1918] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:08:10.560148 13475 catalog_manager.cc:4841] TS de8e6eb4bdc8453ba8eb035d8058fa13 (127.5.198.194:45535): tablet 6b5e7a533af24c7b8c6469f096b30807 (table test-table [id=5234f92c0b5545f38273dc13ddc7a654]) successfully deleted
I20260929 01:08:10.561195 14065 ts_tablet_manager.cc:1918] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:08:10.561403 14064 log.cc:1196] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/6b5e7a533af24c7b8c6469f096b30807
I20260929 01:08:10.561833 14064 ts_tablet_manager.cc:1939] T 6b5e7a533af24c7b8c6469f096b30807 P f1414d21b20f4c1aa99853821d1e0d21: Deleting consensus metadata
I20260929 01:08:10.561982 14065 log.cc:1196] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/6b5e7a533af24c7b8c6469f096b30807
I20260929 01:08:10.563397 14065 ts_tablet_manager.cc:1939] T 6b5e7a533af24c7b8c6469f096b30807 P df78540f1e544a9da2dcc479af683031: Deleting consensus metadata
I20260929 01:08:10.563362 13474 catalog_manager.cc:4841] TS f1414d21b20f4c1aa99853821d1e0d21 (127.5.198.193:36481): tablet 6b5e7a533af24c7b8c6469f096b30807 (table test-table [id=5234f92c0b5545f38273dc13ddc7a654]) successfully deleted
I20260929 01:08:10.564359 13475 catalog_manager.cc:4841] TS df78540f1e544a9da2dcc479af683031 (127.5.198.195:44017): tablet 6b5e7a533af24c7b8c6469f096b30807 (table test-table [id=5234f92c0b5545f38273dc13ddc7a654]) successfully deleted
Sep 29 01:08:10 dist-test-slave-41cl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.198.193@KRBTEST.COM
I20260929 01:08:10.580528 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:10.548633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.5.198.193:36481)
0929 01:08:10.566587 (+ 17954us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:10.566592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:10.566886 (+ 294us) server_negotiation.cc:407] Connection header received
0929 01:08:10.567095 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:10.567100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:10.567183 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:10.567306 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:10.568549 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.569209 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.570395 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.570803 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.570936 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:10.574875 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:10.574899 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:10.574915 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:10.574949 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:10.577853 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:10.578510 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:10.578517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:10.578520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:10.578650 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:10.579024 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:10.579033 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:10.579035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:10.579360 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:10.579670 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:10.579930 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 01:08:10.580146 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17848,"thread_start_us":17937,"threads_started":1}
I20260929 01:08:10.581194 13693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:10.680239 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 13562
I20260929 01:08:10.699110 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 13702
Sep 29 01:08:10 dist-test-slave-41cl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.198.194@KRBTEST.COM
I20260929 01:08:10.714007 14112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:10.699866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53006 (local address 127.5.198.194:45535)
0929 01:08:10.700316 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:10.700320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:10.700336 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:08:10.700388 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:10.700391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:10.700455 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:10.700603 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:10.701977 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.702708 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.703993 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.704373 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.704606 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:10.708674 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:10.708688 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:10.708703 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:10.708748 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:10.711362 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:10.711968 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:10.711975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:10.711977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:10.712110 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:10.712605 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:10.712614 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:10.712620 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:10.713026 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:10.713251 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:10.713544 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 01:08:10.713711 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":254,"threads_started":1}
I20260929 01:08:10.714742 13832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:10.835213 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 13702
I20260929 01:08:10.857909 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 13841
Sep 29 01:08:10 dist-test-slave-41cl krb5kdc[13445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.198.195@KRBTEST.COM
I20260929 01:08:10.872780 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:10.858780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46724 (local address 127.5.198.195:44017)
0929 01:08:10.858952 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:10.858957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:10.858990 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:08:10.859043 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:10.859046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:10.859113 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:10.859200 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:10.860556 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.861468 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.862895 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:10.863292 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:10.863410 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:10.867757 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:10.867771 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:10.867787 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:10.867821 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:10.870522 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:10.871111 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:10.871118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:10.871120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:10.871232 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:10.871563 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:10.871566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:10.871568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:10.871901 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:10.872144 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:10.872317 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 01:08:10.872543 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:08:10.873453 13971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:11.017956 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 13841
I20260929 01:08:11.043655 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 13458
I20260929 01:08:11.044883 13530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:11.157586 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 13458
2026-09-29T01:08:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7098 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:08:11 dist-test-slave-41cl krb5kdc[14119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:08:11 dist-test-slave-41cl krb5kdc[14119](info): set up 2 sockets
Sep 29 01:08:11 dist-test-slave-41cl krb5kdc[14119](info): commencing operation
krb5kdc: starting...
W20260929 01:08:12.261610 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:08:12 dist-test-slave-41cl krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:08:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:08:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:08:12.476920 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:42707
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:42707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:08:12.791620 14132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:12.791877 14132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:12.792033 14132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:12.799068 14132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:08:12.799163 14132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:12.799211 14132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:12.799312 14132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:08:12.799402 14132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:08:12.809051 14132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:42707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:42707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:12.810249 14132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:12.811501 14132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:12.822952 14137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:12.823001 14138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:12.823362 14132 server_base.cc:1034] running on GCE node
W20260929 01:08:12.823611 14140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:12.824343 14132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:12.825708 14132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:12.826912 14132 hybrid_clock.cc:648] HybridClock initialized: now 1790644092826869 us; error 34 us; skew 500 ppm
I20260929 01:08:12.827277 14132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T01:08:12Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1733 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:08:12 dist-test-slave-41cl krb5kdc[14146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:08:12 dist-test-slave-41cl krb5kdc[14146](info): set up 2 sockets
Sep 29 01:08:12 dist-test-slave-41cl krb5kdc[14146](info): commencing operation
krb5kdc: starting...
W20260929 01:08:14.009486 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:08:14 dist-test-slave-41cl krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:08:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:08:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:08:14.221017 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:37093
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:37093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:08:14.533598 14159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:14.533949 14159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:14.534099 14159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:14.541205 14159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:08:14.541348 14159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:14.541455 14159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:14.541563 14159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:08:14.541649 14159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:08:14.551153 14159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:37093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:37093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:14.552498 14159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:14.553752 14159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:14.565290 14164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:14.565322 14167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:14.565501 14165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:14.566454 14159 server_base.cc:1034] running on GCE node
I20260929 01:08:14.567183 14159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:14.568596 14159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:14.569794 14159 hybrid_clock.cc:648] HybridClock initialized: now 1790644094569737 us; error 56 us; skew 500 ppm
I20260929 01:08:14.570122 14159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:14 dist-test-slave-41cl krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644094, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:14.581336 14159 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:08:14.583104 14159 webserver.cc:460] Webserver started at http://127.5.198.254:39655/ using document root <none> and password file <none>
I20260929 01:08:14.583865 14159 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:14.584022 14159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:14.584360 14159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:14.587630 14159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "46d6971c979f40caae60f8934726a188"
format_stamp: "Formatted at 2026-09-29 01:08:14 on dist-test-slave-41cl"
server_key: "e0f25d67c8faa3a09388b23625efc8d3"
server_key_iv: "bb85cfc51aeb97b353a69393af7488f4"
server_key_version: "encryptionkey@0"
I20260929 01:08:14.588531 14159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "46d6971c979f40caae60f8934726a188"
format_stamp: "Formatted at 2026-09-29 01:08:14 on dist-test-slave-41cl"
server_key: "e0f25d67c8faa3a09388b23625efc8d3"
server_key_iv: "bb85cfc51aeb97b353a69393af7488f4"
server_key_version: "encryptionkey@0"
I20260929 01:08:14.594758 14159 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:08:14.598457 14174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:14.599829 14159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:08:14.600004 14159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "46d6971c979f40caae60f8934726a188"
format_stamp: "Formatted at 2026-09-29 01:08:14 on dist-test-slave-41cl"
server_key: "e0f25d67c8faa3a09388b23625efc8d3"
server_key_iv: "bb85cfc51aeb97b353a69393af7488f4"
server_key_version: "encryptionkey@0"
I20260929 01:08:14.600209 14159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:14.624085 14159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:14.628266 14159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:14.628612 14159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:14.647277 14159 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:37093
I20260929 01:08:14.647312 14225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:37093 every 8 connection(s)
I20260929 01:08:14.648700 14159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:08:14.650655 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 14159
I20260929 01:08:14.650870 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:08:14.651690 5915 external_mini_cluster.cc:1428] Setting key cad8774de2d0898ab9a2981c0fc5e2f9
I20260929 01:08:14.653234 14226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 01:08:14 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:08:14.665803 14226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188: Bootstrap starting.
I20260929 01:08:14.669301 14226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:14.670344 14226 log.cc:824] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188: Log is configured to *not* fsync() on all Append() calls
I20260929 01:08:14.672925 14229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:14.653738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.5.198.254:37093)
0929 01:08:14.654467 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:14.654479 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:14.654519 (+ 40us) server_negotiation.cc:407] Connection header received
0929 01:08:14.656137 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:14.656157 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:14.656654 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:14.657049 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:14.658445 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:14.659771 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:14.661236 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:14.661870 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:14.662061 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:14.666103 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:14.666110 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:14.666123 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:14.666154 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:14.669110 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:14.669817 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:14.669825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:14.669840 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:14.669987 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:14.670348 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:14.670353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:14.670363 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:14.670887 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:14.671211 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:14.671587 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 01:08:14.671955 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":295,"threads_started":1}
I20260929 01:08:14.673482 14226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188: No bootstrap required, opened a new log
I20260929 01:08:14.682472 14226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d6971c979f40caae60f8934726a188" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37093 } }
I20260929 01:08:14.682868 14226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:14.682982 14226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46d6971c979f40caae60f8934726a188, State: Initialized, Role: FOLLOWER
I20260929 01:08:14.683555 14226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d6971c979f40caae60f8934726a188" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37093 } }
I20260929 01:08:14.683805 14226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:08:14.683961 14226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:08:14.684145 14226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:14.686923 14226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d6971c979f40caae60f8934726a188" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37093 } }
I20260929 01:08:14.687438 14226 leader_election.cc:302] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46d6971c979f40caae60f8934726a188; no voters:
I20260929 01:08:14.688050 14226 leader_election.cc:288] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:08:14.688190 14231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:08:14.688920 14231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 1 LEADER]: Becoming Leader. State: Replica: 46d6971c979f40caae60f8934726a188, State: Running, Role: LEADER
I20260929 01:08:14.689427 14231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d6971c979f40caae60f8934726a188" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37093 } }
I20260929 01:08:14.690198 14226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:08:14.692405 14232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46d6971c979f40caae60f8934726a188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d6971c979f40caae60f8934726a188" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37093 } } }
I20260929 01:08:14.692399 14233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46d6971c979f40caae60f8934726a188. Latest consensus state: current_term: 1 leader_uuid: "46d6971c979f40caae60f8934726a188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d6971c979f40caae60f8934726a188" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37093 } } }
I20260929 01:08:14.692929 14232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:14.692943 14233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:14.694859 14238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:08:14.700532 14238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:08:14.709318 14238 catalog_manager.cc:1347] Generated new cluster ID: f94bf427a90b4fa49bd9edb870a205c3
I20260929 01:08:14.709448 14238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:08:14.731115 14238 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:08:14.732754 14238 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:08:14.739497 14238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188: Generated new TSK 0
I20260929 01:08:14.740305 14238 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:08:14.800201 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37093
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:08:15.127086 14252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:15.127349 14252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:15.127446 14252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:15.135325 14252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:15.135442 14252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:15.135581 14252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:08:15.145362 14252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:15.146579 14252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:15.147820 14252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:15.160674 14260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:15.160703 14258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:15.161089 14257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:15.162294 14252 server_base.cc:1034] running on GCE node
I20260929 01:08:15.162948 14252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:15.163959 14252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:15.165211 14252 hybrid_clock.cc:648] HybridClock initialized: now 1790644095165159 us; error 72 us; skew 500 ppm
I20260929 01:08:15.165571 14252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:15 dist-test-slave-41cl krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644095, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:15.168910 14252 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:08:15.170312 14252 webserver.cc:460] Webserver started at http://127.5.198.193:42833/ using document root <none> and password file <none>
I20260929 01:08:15.170971 14252 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:15.171111 14252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:15.171427 14252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:15.174719 14252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "95ed38d98886422a92fc94baaa475fb3"
format_stamp: "Formatted at 2026-09-29 01:08:15 on dist-test-slave-41cl"
server_key: "f0eeb5a3d973222b3bae7c69829853ee"
server_key_iv: "0d1ad8017941fb850180075b8b59ed26"
server_key_version: "encryptionkey@0"
I20260929 01:08:15.175597 14252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "95ed38d98886422a92fc94baaa475fb3"
format_stamp: "Formatted at 2026-09-29 01:08:15 on dist-test-slave-41cl"
server_key: "f0eeb5a3d973222b3bae7c69829853ee"
server_key_iv: "0d1ad8017941fb850180075b8b59ed26"
server_key_version: "encryptionkey@0"
I20260929 01:08:15.180691 14252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 01:08:15.184165 14267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:15.185575 14252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:08:15.185745 14252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "95ed38d98886422a92fc94baaa475fb3"
format_stamp: "Formatted at 2026-09-29 01:08:15 on dist-test-slave-41cl"
server_key: "f0eeb5a3d973222b3bae7c69829853ee"
server_key_iv: "0d1ad8017941fb850180075b8b59ed26"
server_key_version: "encryptionkey@0"
I20260929 01:08:15.185945 14252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:15.213688 14252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:15.217549 14252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:15.217834 14252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:15.218763 14252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:15.220005 14252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:15.220103 14252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:15.220252 14252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:15.220331 14252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:15.253034 14252 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:43151
I20260929 01:08:15.253126 14379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:43151 every 8 connection(s)
I20260929 01:08:15.254354 14252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:08:15.264937 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 14252
I20260929 01:08:15.265151 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:08:15.265991 5915 external_mini_cluster.cc:1428] Setting key dac49f89f359080111845643a8b279c4
Sep 29 01:08:15 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644095, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:08:15.274612 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:15.257045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:46571 (local address 127.5.198.254:37093)
0929 01:08:15.257465 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:15.257470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:15.258439 (+ 969us) server_negotiation.cc:407] Connection header received
0929 01:08:15.260392 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:15.260398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:15.260525 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:15.260656 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:15.262235 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:15.263000 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:15.264859 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:15.265351 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:15.265458 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:15.269288 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:15.269293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:15.269296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:15.269324 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:15.271777 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:15.272445 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:15.272452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:15.272459 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:15.272611 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:15.273014 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:15.273021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:15.273024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:15.273263 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:15.273413 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:15.274063 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 01:08:15.274273 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":218,"threads_started":1}
I20260929 01:08:15.275677 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:15.257524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:46571)
0929 01:08:15.258265 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:15.258318 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:15.260122 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:15.260910 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:15.260919 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:15.261451 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:15.262047 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:15.262065 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:15.263181 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:15.263185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:15.264691 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:15.264707 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:15.265585 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:15.265594 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:15.265937 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:15.266678 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:15.266704 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:15.269085 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:15.271915 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:15.271924 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:15.271939 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:15.272329 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:15.272717 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:15.272723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:15.272727 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:15.272907 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:15.273383 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:15.273391 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:15.273716 (+ 325us) client_negotiation.cc:769] Sending connection context
0929 01:08:15.273965 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 01:08:15.274476 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":289,"threads_started":1}
I20260929 01:08:15.277209 14380 heartbeater.cc:347] Connected to a master server at 127.5.198.254:37093
I20260929 01:08:15.277498 14380 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:15.278153 14380 heartbeater.cc:511] Master 127.5.198.254:37093 requested a full tablet report, sending...
I20260929 01:08:15.280382 14191 ts_manager.cc:194] Registered new tserver with Master: 95ed38d98886422a92fc94baaa475fb3 (127.5.198.193:43151)
I20260929 01:08:15.282759 14191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:46571
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:08:15.316829 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37093
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:08:15.634495 14386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:15.634824 14386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:15.635004 14386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:15.642089 14386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:15.642236 14386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:15.642421 14386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:08:15.651854 14386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:15.653061 14386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:15.654287 14386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:15.667212 14391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:15.667207 14394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:15.667239 14392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:15.669834 14386 server_base.cc:1034] running on GCE node
I20260929 01:08:15.670388 14386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:15.671288 14386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:15.672556 14386 hybrid_clock.cc:648] HybridClock initialized: now 1790644095672518 us; error 56 us; skew 500 ppm
I20260929 01:08:15.672857 14386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:15 dist-test-slave-41cl krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644095, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:15.675971 14386 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:08:15.677835 14386 webserver.cc:460] Webserver started at http://127.5.198.194:42665/ using document root <none> and password file <none>
I20260929 01:08:15.678570 14386 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:15.678723 14386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:15.679071 14386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:15.682081 14386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "9c83cd3624124531af276297f0eba866"
format_stamp: "Formatted at 2026-09-29 01:08:15 on dist-test-slave-41cl"
server_key: "ef72222ad602dae42fb4eadd98a0e799"
server_key_iv: "7728b3e65a9bbadffea6f73e6380eafa"
server_key_version: "encryptionkey@0"
I20260929 01:08:15.682842 14386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "9c83cd3624124531af276297f0eba866"
format_stamp: "Formatted at 2026-09-29 01:08:15 on dist-test-slave-41cl"
server_key: "ef72222ad602dae42fb4eadd98a0e799"
server_key_iv: "7728b3e65a9bbadffea6f73e6380eafa"
server_key_version: "encryptionkey@0"
I20260929 01:08:15.687848 14386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 01:08:15.691761 14401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:15.693208 14386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:08:15.693391 14386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "9c83cd3624124531af276297f0eba866"
format_stamp: "Formatted at 2026-09-29 01:08:15 on dist-test-slave-41cl"
server_key: "ef72222ad602dae42fb4eadd98a0e799"
server_key_iv: "7728b3e65a9bbadffea6f73e6380eafa"
server_key_version: "encryptionkey@0"
I20260929 01:08:15.693593 14386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:15.712309 14386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:15.716498 14386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:15.716782 14386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:15.717731 14386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:15.719046 14386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:15.719149 14386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:15.719300 14386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:15.719370 14386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:15.753227 14386 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:43525
I20260929 01:08:15.753311 14513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:43525 every 8 connection(s)
I20260929 01:08:15.754426 14386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:08:15.757054 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 14386
I20260929 01:08:15.757220 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:08:15.757745 5915 external_mini_cluster.cc:1428] Setting key c5580800fc28f0ce059ec0f7b28acdb3
Sep 29 01:08:15 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644095, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:08:15.774112 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:15.757046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:59153 (local address 127.5.198.254:37093)
0929 01:08:15.757208 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:15.757212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:15.758660 (+ 1448us) server_negotiation.cc:407] Connection header received
0929 01:08:15.760827 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:15.760832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:15.760910 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:15.761029 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:15.762559 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:15.763241 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:15.764791 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:15.765223 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:15.765354 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:15.768784 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:15.768790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:15.768795 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:15.768833 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:15.771301 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:15.771978 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:15.771983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:15.771987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:15.772108 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:15.772528 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:15.772533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:15.772537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:15.772782 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:15.772932 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:15.773561 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 01:08:15.773771 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:08:15.775023 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:15.757661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:59153)
0929 01:08:15.758489 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:15.758543 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:15.760591 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:15.761246 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:15.761254 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:15.761826 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:15.762405 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:15.762423 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:15.763393 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:15.763398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:15.764659 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:15.764670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:15.765436 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:15.765444 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:15.765663 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:15.766317 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:15.766342 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:15.768586 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:15.771436 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:15.771447 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:15.771462 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:15.771848 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:15.772213 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:15.772219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:15.772223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:15.772395 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:15.772904 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:15.772911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:15.773219 (+ 308us) client_negotiation.cc:769] Sending connection context
0929 01:08:15.773465 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 01:08:15.773964 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":291,"threads_started":1}
I20260929 01:08:15.776631 14514 heartbeater.cc:347] Connected to a master server at 127.5.198.254:37093
I20260929 01:08:15.776938 14514 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
I20260929 01:08:15.777614 14514 heartbeater.cc:511] Master 127.5.198.254:37093 requested a full tablet report, sending...
I20260929 01:08:15.778960 14191 ts_manager.cc:194] Registered new tserver with Master: 9c83cd3624124531af276297f0eba866 (127.5.198.194:43525)
I20260929 01:08:15.779884 14191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:59153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:08:15.807190 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37093
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:08:16.124531 14519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:16.124789 14519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:16.124879 14519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:16.131702 14519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:16.131791 14519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:16.131896 14519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:08:16.141031 14519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:16.142148 14519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:16.143473 14519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:16.156123 14524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:16.156211 14527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:16.156189 14525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:16.157099 14519 server_base.cc:1034] running on GCE node
I20260929 01:08:16.157668 14519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:16.158568 14519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:16.159785 14519 hybrid_clock.cc:648] HybridClock initialized: now 1790644096159749 us; error 50 us; skew 500 ppm
I20260929 01:08:16.160081 14519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:16 dist-test-slave-41cl krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644096, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:16.163095 14519 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:08:16.164592 14519 webserver.cc:460] Webserver started at http://127.5.198.195:37371/ using document root <none> and password file <none>
I20260929 01:08:16.165282 14519 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:16.165422 14519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:16.165721 14519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:16.168453 14519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "ed7fb6ff80b04027abc7b6c57cccb0b5"
format_stamp: "Formatted at 2026-09-29 01:08:16 on dist-test-slave-41cl"
server_key: "a2e82aac638fee9c47c03ddcc4b8c515"
server_key_iv: "5e51d896527c1125cba473268e0b9f42"
server_key_version: "encryptionkey@0"
I20260929 01:08:16.169279 14519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "ed7fb6ff80b04027abc7b6c57cccb0b5"
format_stamp: "Formatted at 2026-09-29 01:08:16 on dist-test-slave-41cl"
server_key: "a2e82aac638fee9c47c03ddcc4b8c515"
server_key_iv: "5e51d896527c1125cba473268e0b9f42"
server_key_version: "encryptionkey@0"
I20260929 01:08:16.174119 14519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:08:16.177577 14534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:16.178879 14519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:08:16.179068 14519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "ed7fb6ff80b04027abc7b6c57cccb0b5"
format_stamp: "Formatted at 2026-09-29 01:08:16 on dist-test-slave-41cl"
server_key: "a2e82aac638fee9c47c03ddcc4b8c515"
server_key_iv: "5e51d896527c1125cba473268e0b9f42"
server_key_version: "encryptionkey@0"
I20260929 01:08:16.179265 14519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:16.196238 14519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:16.200563 14519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:16.200839 14519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:16.201716 14519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:16.202991 14519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:16.203117 14519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:16.203286 14519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:16.203358 14519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:16.235105 14519 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:41991
I20260929 01:08:16.235210 14646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:41991 every 8 connection(s)
I20260929 01:08:16.236359 14519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:08:16.245852 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 14519
I20260929 01:08:16.246011 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:08:16.246547 5915 external_mini_cluster.cc:1428] Setting key 88c2008649a5c4b66dea17f6ee92ef3f
Sep 29 01:08:16 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644096, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:08:16.256788 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:16.238920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:51925 (local address 127.5.198.254:37093)
0929 01:08:16.239135 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:16.239141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:16.240307 (+ 1166us) server_negotiation.cc:407] Connection header received
0929 01:08:16.242092 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:16.242097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:16.242196 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:16.242293 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:16.243746 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:16.244656 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:16.246611 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:16.247054 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:16.247187 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:16.250858 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:16.250863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:16.250866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:16.250897 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:16.253470 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:16.254303 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:16.254308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:16.254310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:16.254461 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:16.254940 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:16.254943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:16.254946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:16.255212 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:16.255326 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:16.256206 (+ 880us) server_negotiation.cc:299] Negotiation successful
0929 01:08:16.256409 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 01:08:16.257874 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:16.239436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:51925)
0929 01:08:16.240135 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:16.240187 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:16.241784 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:16.242492 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:16.242501 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:16.243024 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:16.243566 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:16.243583 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:16.244800 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:16.244804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:16.246454 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:16.246468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:16.247320 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:16.247328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:16.247625 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:16.248329 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:16.248384 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:16.250688 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:16.253606 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:16.253617 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:16.253635 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:16.254167 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:16.254582 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:16.254590 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:16.254594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:16.254815 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:16.255343 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:16.255351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:16.255727 (+ 376us) client_negotiation.cc:769] Sending connection context
0929 01:08:16.256043 (+ 316us) client_negotiation.cc:240] Negotiation successful
0929 01:08:16.256579 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":265,"threads_started":1}
I20260929 01:08:16.259163 14647 heartbeater.cc:347] Connected to a master server at 127.5.198.254:37093
I20260929 01:08:16.259454 14647 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:16.260200 14647 heartbeater.cc:511] Master 127.5.198.254:37093 requested a full tablet report, sending...
I20260929 01:08:16.261884 14190 ts_manager.cc:194] Registered new tserver with Master: ed7fb6ff80b04027abc7b6c57cccb0b5 (127.5.198.195:41991)
I20260929 01:08:16.262746 14190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:51925
I20260929 01:08:16.273638 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:08:16.285739 14380 heartbeater.cc:503] Master 127.5.198.254:37093 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.193.keytab.
Entry for principal oryx/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.193.keytab.
WARNING: no policy specified for oryx/127.5.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.194.keytab.
Entry for principal oryx/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.194.keytab.
WARNING: no policy specified for oryx/127.5.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.195.keytab.
Entry for principal oryx/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.195.keytab.
I20260929 01:08:16.407251 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 14252
Sep 29 01:08:16 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:08:16.423810 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:16.408418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50102 (local address 127.5.198.193:43151)
0929 01:08:16.409039 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:16.409046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:16.409074 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:08:16.409132 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:16.409139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:16.409351 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:16.409486 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:16.410720 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:16.411815 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:16.413654 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:16.414631 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:16.414746 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:16.418717 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:16.418728 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:16.418742 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:16.418778 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:16.421479 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:16.422117 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:16.422124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:16.422126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:16.422244 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:16.422568 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:16.422571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:16.422572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:16.422880 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:16.423086 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:16.423328 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 01:08:16.423507 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":305,"threads_started":1}
I20260929 01:08:16.425113 14375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:16.507105 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 14252
I20260929 01:08:16.523111 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 14386
Sep 29 01:08:16 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:08:16.538501 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:16.523949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.5.198.194:43525)
0929 01:08:16.524495 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:16.524506 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:16.524537 (+ 31us) server_negotiation.cc:407] Connection header received
0929 01:08:16.524605 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:16.524612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:16.524815 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:16.524942 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:16.526107 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:16.527112 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:16.528266 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:16.528948 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:16.529051 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:16.532868 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:16.532883 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:16.532897 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:16.532927 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:16.535643 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:16.536286 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:16.536293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:16.536295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:16.536412 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:16.537028 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:16.537032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:16.537034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:16.537409 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:16.537646 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:16.538019 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 01:08:16.538209 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":250,"threads_started":1}
I20260929 01:08:16.539781 14509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:16.622617 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 14386
I20260929 01:08:16.639034 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 14519
Sep 29 01:08:16 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:08:16.655435 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:16.640152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.5.198.195:41991)
0929 01:08:16.640819 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:16.640826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:16.640855 (+ 29us) server_negotiation.cc:407] Connection header received
0929 01:08:16.640916 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:16.640922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:16.641119 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:16.641310 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:16.642624 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:16.643683 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:16.645054 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:16.645602 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:16.645745 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:16.649931 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:16.649942 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:16.649956 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:16.649986 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:16.652805 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:16.653431 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:16.653439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:16.653441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:16.653563 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:16.653930 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:16.653936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:16.653939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:16.654261 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:16.654568 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:16.654842 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 01:08:16.655097 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":383,"threads_started":1}
I20260929 01:08:16.656760 14642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:16.739310 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 14519
I20260929 01:08:16.754856 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 14159
I20260929 01:08:16.756089 14221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:16.856894 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 14159
I20260929 01:08:16.877257 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:37093
--webserver_interface=127.5.198.254
--webserver_port=39655
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:37093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:08:17.189697 14664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:17.190006 14664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:17.190171 14664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:17.197167 14664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:08:17.197300 14664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:17.197422 14664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:17.197516 14664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:08:17.197604 14664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:08:17.207036 14664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:37093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:37093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=39655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:17.208323 14664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:17.209643 14664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:17.221024 14669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:17.221098 14672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:17.221113 14670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:17.221912 14664 server_base.cc:1034] running on GCE node
I20260929 01:08:17.222592 14664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:17.223863 14664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:17.225090 14664 hybrid_clock.cc:648] HybridClock initialized: now 1790644097225043 us; error 57 us; skew 500 ppm
I20260929 01:08:17.225432 14664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:17 dist-test-slave-41cl krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644097, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:17.228302 14664 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:08:17.229799 14664 webserver.cc:460] Webserver started at http://127.5.198.254:39655/ using document root <none> and password file <none>
I20260929 01:08:17.230463 14664 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:17.230615 14664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:17.235843 14664 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:08:17.238737 14679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:17.240098 14664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 01:08:17.240276 14664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "46d6971c979f40caae60f8934726a188"
format_stamp: "Formatted at 2026-09-29 01:08:14 on dist-test-slave-41cl"
server_key: "e0f25d67c8faa3a09388b23625efc8d3"
server_key_iv: "bb85cfc51aeb97b353a69393af7488f4"
server_key_version: "encryptionkey@0"
I20260929 01:08:17.241227 14664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:17.266474 14664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:17.270816 14664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:17.271100 14664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:17.290422 14664 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:37093
I20260929 01:08:17.290524 14730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:37093 every 8 connection(s)
I20260929 01:08:17.291891 14664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:08:17.297062 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 14664
I20260929 01:08:17.298002 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:43151
--local_ip_for_outbound_sockets=127.5.198.193
--tserver_master_addrs=127.5.198.254:37093
--webserver_port=42833
--webserver_interface=127.5.198.193
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.198.193
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:08:17.299293 14731 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:08:17.304232 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188: Bootstrap starting.
I20260929 01:08:17.323079 14731 log.cc:824] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188: Log is configured to *not* fsync() on all Append() calls
I20260929 01:08:17.331683 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:08:17.332203 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188: Bootstrap complete.
I20260929 01:08:17.341009 14731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d6971c979f40caae60f8934726a188" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37093 } }
I20260929 01:08:17.341773 14731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46d6971c979f40caae60f8934726a188, State: Initialized, Role: FOLLOWER
I20260929 01:08:17.342487 14731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d6971c979f40caae60f8934726a188" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37093 } }
I20260929 01:08:17.342715 14731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:08:17.342821 14731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:08:17.343011 14731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:08:17.347026 14731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d6971c979f40caae60f8934726a188" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37093 } }
I20260929 01:08:17.347498 14731 leader_election.cc:302] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46d6971c979f40caae60f8934726a188; no voters:
I20260929 01:08:17.348078 14731 leader_election.cc:288] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 01:08:17.348315 14735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:08:17.349397 14735 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [term 2 LEADER]: Becoming Leader. State: Replica: 46d6971c979f40caae60f8934726a188, State: Running, Role: LEADER
I20260929 01:08:17.350072 14731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:08:17.350188 14735 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d6971c979f40caae60f8934726a188" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37093 } }
I20260929 01:08:17.354212 14736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "46d6971c979f40caae60f8934726a188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d6971c979f40caae60f8934726a188" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37093 } } }
I20260929 01:08:17.354635 14736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:17.355218 14737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46d6971c979f40caae60f8934726a188. Latest consensus state: current_term: 2 leader_uuid: "46d6971c979f40caae60f8934726a188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d6971c979f40caae60f8934726a188" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37093 } } }
I20260929 01:08:17.355568 14737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188 [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:17.355572 14745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:08:17.364260 14745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:08:17.365752 14745 catalog_manager.cc:1259] Loaded cluster ID: f94bf427a90b4fa49bd9edb870a205c3
I20260929 01:08:17.365865 14745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:08:17.369939 14745 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:08:17.371481 14745 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 46d6971c979f40caae60f8934726a188: Loaded TSK: 0
I20260929 01:08:17.373008 14745 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 01:08:17.637081 14733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:17.637360 14733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:17.637512 14733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:17.644874 14733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:17.644981 14733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:17.645231 14733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:08:17.655602 14733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.193.keytab
--principal=oryx/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:43151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=42833
--tserver_master_addrs=127.5.198.254:37093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:17.657044 14733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:17.658407 14733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:17.673377 14761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:17.673401 14759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:17.676916 14758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:17.678759 14733 server_base.cc:1034] running on GCE node
I20260929 01:08:17.679554 14733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:17.680724 14733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:17.681972 14733 hybrid_clock.cc:648] HybridClock initialized: now 1790644097681927 us; error 36 us; skew 500 ppm
I20260929 01:08:17.682325 14733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:17 dist-test-slave-41cl krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644097, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:17.686286 14733 init.cc:377] Logged in from keytab as oryx/127.5.198.193@KRBTEST.COM (short username oryx)
I20260929 01:08:17.688088 14733 webserver.cc:460] Webserver started at http://127.5.198.193:42833/ using document root <none> and password file <none>
I20260929 01:08:17.688990 14733 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:17.689129 14733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:17.696240 14733 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 01:08:17.699450 14768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:17.701154 14733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:08:17.701323 14733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "95ed38d98886422a92fc94baaa475fb3"
format_stamp: "Formatted at 2026-09-29 01:08:15 on dist-test-slave-41cl"
server_key: "f0eeb5a3d973222b3bae7c69829853ee"
server_key_iv: "0d1ad8017941fb850180075b8b59ed26"
server_key_version: "encryptionkey@0"
I20260929 01:08:17.702600 14733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:17.724191 14733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:17.728801 14733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:17.729017 14733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:17.729902 14733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:17.731091 14733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:17.731175 14733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:17.731243 14733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:17.731292 14733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:17.764606 14733 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:43151
I20260929 01:08:17.764771 14880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:43151 every 8 connection(s)
I20260929 01:08:17.766047 14733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:08:17.773434 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 14733
I20260929 01:08:17.774077 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:43525
--local_ip_for_outbound_sockets=127.5.198.194
--tserver_master_addrs=127.5.198.254:37093
--webserver_port=42665
--webserver_interface=127.5.198.194
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.198.194
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:08:17 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:17 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 01:08:18.098248 14885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:18.098541 14885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:18.098697 14885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:18.105700 14885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:18.105847 14885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:18.106015 14885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:08:18.115598 14885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.194.keytab
--principal=oryx/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:43525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=42665
--tserver_master_addrs=127.5.198.254:37093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:18.116976 14885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:18.118242 14885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:18.136081 14892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:18.136278 14891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:18.136600 14894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:18.137079 14885 server_base.cc:1034] running on GCE node
I20260929 01:08:18.137647 14885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:18.138631 14885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:18.139942 14885 hybrid_clock.cc:648] HybridClock initialized: now 1790644098139867 us; error 86 us; skew 500 ppm
I20260929 01:08:18.140254 14885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:18 dist-test-slave-41cl krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644098, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:18.143412 14885 init.cc:377] Logged in from keytab as oryx/127.5.198.194@KRBTEST.COM (short username oryx)
I20260929 01:08:18.145108 14885 webserver.cc:460] Webserver started at http://127.5.198.194:42665/ using document root <none> and password file <none>
I20260929 01:08:18.145823 14885 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:18.145961 14885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:18.151861 14885 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 01:08:18.154698 14901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:18.156735 14885 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 01:08:18.156896 14885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "9c83cd3624124531af276297f0eba866"
format_stamp: "Formatted at 2026-09-29 01:08:15 on dist-test-slave-41cl"
server_key: "ef72222ad602dae42fb4eadd98a0e799"
server_key_iv: "7728b3e65a9bbadffea6f73e6380eafa"
server_key_version: "encryptionkey@0"
I20260929 01:08:18.157791 14885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:18.174407 14885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:18.178716 14885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:18.178983 14885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:18.180013 14885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:18.181468 14885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:18.181559 14885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:18.181699 14885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:18.181797 14885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:18.226083 14885 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:43525
I20260929 01:08:18.226133 15013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:43525 every 8 connection(s)
I20260929 01:08:18.227496 14885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:08:18.229672 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 14885
I20260929 01:08:18.230661 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:41991
--local_ip_for_outbound_sockets=127.5.198.195
--tserver_master_addrs=127.5.198.254:37093
--webserver_port=37371
--webserver_interface=127.5.198.195
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.198.195
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:08:18 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:18 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 01:08:18.927786 15018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:18.928210 15018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:18.928380 15018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:18.940061 15018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:18.940212 15018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:18.940482 15018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:08:18.956671 15018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/krb5kdc/oryx_127.5.198.195.keytab
--principal=oryx/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:41991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=37371
--tserver_master_addrs=127.5.198.254:37093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:18.958349 15018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:18.960001 15018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:18.979874 15025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:18.993044 15024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:18.994271 15028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:18.994575 15018 server_base.cc:1034] running on GCE node
I20260929 01:08:18.995357 15018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:18.996516 15018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:18.997795 15018 hybrid_clock.cc:648] HybridClock initialized: now 1790644098997717 us; error 63 us; skew 500 ppm
I20260929 01:08:18.998176 15018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:18 dist-test-slave-41cl krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644098, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:19.001370 15018 init.cc:377] Logged in from keytab as oryx/127.5.198.195@KRBTEST.COM (short username oryx)
I20260929 01:08:19.003050 15018 webserver.cc:460] Webserver started at http://127.5.198.195:37371/ using document root <none> and password file <none>
I20260929 01:08:19.003739 15018 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:19.003894 15018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:19.009562 15018 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:08:19.012171 15034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:19.013602 15018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:08:19.013782 15018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "ed7fb6ff80b04027abc7b6c57cccb0b5"
format_stamp: "Formatted at 2026-09-29 01:08:16 on dist-test-slave-41cl"
server_key: "a2e82aac638fee9c47c03ddcc4b8c515"
server_key_iv: "5e51d896527c1125cba473268e0b9f42"
server_key_version: "encryptionkey@0"
I20260929 01:08:19.014714 15018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:19.029610 15018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:19.033911 15018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:19.034184 15018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:19.035192 15018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:19.036672 15018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:19.036788 15018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:19.036908 15018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:19.037007 15018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:19.077124 15018 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:41991
I20260929 01:08:19.077164 15146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:41991 every 8 connection(s)
I20260929 01:08:19.078634 15018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
W20260929 01:08:19.080950 14884 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.300s user 0.006s sys 0.012s
I20260929 01:08:19.083166 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:17.769220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:34137)
0929 01:08:17.770052 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:17.770095 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:17.771694 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:17.773113 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:17.773123 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:17.773762 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:17.774400 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:17.774423 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:17.776847 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:17.776851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:17.778106 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:17.778118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:17.779554 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:17.779559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:17.779741 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:17.780413 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:17.780433 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:19.082097 (+1301664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":355,"threads_started":1}
W20260929 01:08:19.083223 14883 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:17.769121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:34137 (local address 127.5.198.254:37093)
0929 01:08:17.769858 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:17.769874 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:17.770210 (+ 336us) server_negotiation.cc:407] Connection header received
0929 01:08:17.772213 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:17.772226 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:17.772590 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:17.772962 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:17.774623 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:17.776688 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:17.778251 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:17.779347 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:17.779555 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:19.082467 (+1302912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:34137: BlockingRecv error: recv got EOF from 127.5.198.193:34137 (error 108)
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":227,"threads_started":1}
W20260929 01:08:19.084334 14881 heartbeater.cc:644] Failed to heartbeat to 127.5.198.254:37093 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.198.254:37093: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
I20260929 01:08:19.087414 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 15018
Sep 29 01:08:19 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:19 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:19.107729 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:19.085295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:42777)
0929 01:08:19.085642 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:19.085670 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:19.085822 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:19.089588 (+ 3766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:19.089598 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:19.090247 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:19.090609 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:19.090626 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.095770 (+ 5144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.095779 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:19.098118 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:19.098140 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.101918 (+ 3778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.101927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:19.102141 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:19.102886 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:19.102910 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:19.107236 (+ 4326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186}
W20260929 01:08:19.110716 15150 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:19.085378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:42777 (local address 127.5.198.254:37093)
0929 01:08:19.089068 (+ 3690us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:19.089078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:19.089105 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:08:19.089193 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:19.089201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:19.089324 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:19.090819 (+ 1495us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:19.091161 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.092941 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.098414 (+ 5473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.099409 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.104164 (+ 4755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:19.110184 (+ 6020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:42777: BlockingRecv error: recv got EOF from 127.5.198.193:42777 (error 108)
Metrics: {"server-negotiator.queue_time_us":3537}
Sep 29 01:08:19 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:19 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:19 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:19 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:19.125067 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:19.109108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:44007)
0929 01:08:19.109426 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:19.109450 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:19.109572 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:19.111682 (+ 2110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:19.111690 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:19.112251 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:19.112706 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:19.112724 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.114765 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.114774 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:19.117083 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:19.117104 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.118717 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.118727 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:19.119003 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:19.119780 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:19.119814 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:19.124808 (+ 4994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174}
W20260929 01:08:19.125797 14881 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:08:19.126431 15150 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:19.109158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:44007 (local address 127.5.198.254:37093)
0929 01:08:19.111002 (+ 1844us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:19.111010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:19.111034 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:08:19.111101 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:19.111108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:19.111200 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:19.111338 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:19.112926 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.114536 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.117330 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.118288 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.118448 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:19.126020 (+ 7572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:44007: BlockingRecv error: recv got EOF from 127.5.198.193:44007 (error 108)
Metrics: {"server-negotiator.queue_time_us":1709,"thread_start_us":6512,"threads_started":1}
I20260929 01:08:19.126981 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:19.088950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.5.198.254:37093)
0929 01:08:19.101236 (+ 12286us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:19.101240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:19.101256 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:08:19.101294 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:19.101298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:19.101348 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:19.101471 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:19.104371 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.106918 (+ 2547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.109392 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.110420 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.110570 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:19.119410 (+ 8840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:19.119423 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:19.119462 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:19.119518 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:19.122622 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:19.123932 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:19.123939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:19.123945 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:19.124118 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:19.124722 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:19.124729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:19.124735 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:19.125356 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:19.125654 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:19.126060 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 01:08:19.126417 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12188,"thread_start_us":12698,"threads_started":1}
W20260929 01:08:19.398033 15017 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.145s user 0.007s sys 0.007s
W20260929 01:08:19.399657 15016 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:18.230175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:37437 (local address 127.5.198.254:37093)
0929 01:08:18.243762 (+ 13587us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:18.243774 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:18.244832 (+ 1058us) server_negotiation.cc:407] Connection header received
0929 01:08:18.246841 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:18.246846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:18.246930 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:18.247061 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:18.248653 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:18.249389 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:18.251226 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:18.251680 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:18.251819 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:19.399413 (+1147594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:37437: BlockingRecv error: recv got EOF from 127.5.198.194:37437 (error 108)
Metrics: {"server-negotiator.queue_time_us":13372,"thread_start_us":13215,"threads_started":1}
I20260929 01:08:19.400933 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:18.230883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:37437)
0929 01:08:18.244192 (+ 13309us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:18.244259 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:18.246560 (+ 2301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:18.247312 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:18.247321 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:18.247852 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:18.248430 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:18.248447 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:18.249545 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:18.249551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:18.251051 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:18.251064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:18.251913 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:18.251923 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:18.252149 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:18.252880 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:18.252902 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:19.399277 (+1146375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":12982,"thread_start_us":12736,"threads_started":1}
W20260929 01:08:19.401860 15014 heartbeater.cc:644] Failed to heartbeat to 127.5.198.254:37093 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.198.254:37093: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Sep 29 01:08:19 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:19 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 01:08:19.412292 15016 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:19.402633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:42297 (local address 127.5.198.254:37093)
0929 01:08:19.402813 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:19.402818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:19.402835 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:08:19.403033 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:19.403038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:19.403110 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:19.403205 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:19.403883 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.404709 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.405965 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.406366 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.406490 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:19.412136 (+ 5646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:42297: BlockingRecv error: recv got EOF from 127.5.198.194:42297 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:08:19.412431 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:19.402543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:42297)
0929 01:08:19.402723 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:19.402742 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:19.402895 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:19.403230 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:19.403234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:19.403591 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:19.403738 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:19.403746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.404855 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.404860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:19.405802 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:19.405813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.406541 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.406547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:19.406717 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:19.407529 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:19.407546 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:19.412019 (+ 4473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
Sep 29 01:08:19 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:19 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:19.425109 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:19.413212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:34511)
0929 01:08:19.413497 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:19.413522 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:19.413657 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:19.414034 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:19.414041 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:19.414590 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:19.414878 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:19.414894 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.415991 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.415999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:19.418389 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:19.418411 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.419156 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.419163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:19.419340 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:19.420114 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:19.420131 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:19.424664 (+ 4533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148}
W20260929 01:08:19.425137 15016 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:19.413342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:34511 (local address 127.5.198.254:37093)
0929 01:08:19.413487 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:19.413491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:19.413597 (+ 106us) server_negotiation.cc:407] Connection header received
0929 01:08:19.413776 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:19.413779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:19.413873 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:19.413949 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:19.415072 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.415832 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.418560 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.418988 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.419142 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:19.424881 (+ 5739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:34511: BlockingRecv error: recv got EOF from 127.5.198.194:34511 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 01:08:19.426376 15014 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 01:08:20 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:20 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:20.147707 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:20.126397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:36793)
0929 01:08:20.129931 (+ 3534us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:20.129947 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:20.130057 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:20.130633 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:20.130641 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:20.131193 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:20.131343 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:20.131352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:20.133330 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:20.133334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:20.134284 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:20.134295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:20.135665 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:20.135670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:20.135805 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:20.143037 (+ 7232us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:20.143058 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:20.147448 (+ 4390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3418,"thread_start_us":3261,"threads_started":1}
W20260929 01:08:20.148221 15156 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:20.126559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:36793 (local address 127.5.198.254:37093)
0929 01:08:20.129757 (+ 3198us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:20.129767 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:20.130038 (+ 271us) server_negotiation.cc:407] Connection header received
0929 01:08:20.130185 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:20.130193 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:20.130333 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:20.130501 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:20.131496 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:20.133162 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:20.134449 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:20.135474 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:20.135643 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:20.147811 (+ 12168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:36793: BlockingRecv error: recv got EOF from 127.5.198.193:36793 (error 108)
Metrics: {"server-negotiator.queue_time_us":3017,"thread_start_us":2867,"threads_started":1}
W20260929 01:08:20.246197 15149 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.133s user 0.002s sys 0.013s
W20260929 01:08:20.247370 14883 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:19.083230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:38651 (local address 127.5.198.254:37093)
0929 01:08:19.083579 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:19.083585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:19.094088 (+ 10503us) server_negotiation.cc:407] Connection header received
0929 01:08:19.102142 (+ 8054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:19.102150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:19.102253 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:19.102356 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:19.105760 (+ 3404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.106918 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.110856 (+ 3938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.111705 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.111826 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:20.247091 (+1135265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.195:38651: BlockingRecv error: recv got EOF from 127.5.198.195:38651 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":5532,"threads_started":1}
I20260929 01:08:20.247977 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:19.082565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:38651)
0929 01:08:19.093892 (+ 11327us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:19.093965 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:19.101684 (+ 7719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:19.103126 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:19.103144 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:19.104233 (+ 1089us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:19.105462 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:19.105497 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.107156 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.107163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:19.109237 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:19.109251 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:19.112202 (+ 2951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:19.112211 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:19.112631 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:19.113560 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:19.113601 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:20.246936 (+1133335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":10875,"thread_start_us":10579,"threads_started":1}
W20260929 01:08:20.249009 15147 heartbeater.cc:644] Failed to heartbeat to 127.5.198.254:37093 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.198.254:37093: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Sep 29 01:08:20 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:20 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:20.258580 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:20.249732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:47119)
0929 01:08:20.249949 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:20.250009 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:20.250125 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:20.250518 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:20.250522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:20.250925 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:20.251084 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:20.251093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:20.252126 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:20.252132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:20.253142 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:20.253153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:20.253866 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:20.253872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:20.254036 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:20.254658 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:20.254676 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:20.258357 (+ 3681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260929 01:08:20.258826 14883 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:20.249883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:47119 (local address 127.5.198.254:37093)
0929 01:08:20.250112 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:20.250117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:20.250135 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:08:20.250270 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:20.250277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:20.250389 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:20.250490 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:20.251226 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:20.251982 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:20.253290 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:20.253687 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:20.253818 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:20.258562 (+ 4744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.195:47119: BlockingRecv error: recv got EOF from 127.5.198.195:47119 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Sep 29 01:08:20 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:20 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:20.267740 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:20.259341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:36099)
0929 01:08:20.259515 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:20.259535 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:20.259635 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:20.260034 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:20.260041 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:20.260364 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:20.260569 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:20.260583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:20.261691 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:20.261696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:20.262836 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:20.262847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:20.263533 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:20.263539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:20.263705 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:20.264273 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:20.264291 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:20.267466 (+ 3175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 01:08:20.267910 14883 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:20.259523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:36099 (local address 127.5.198.254:37093)
0929 01:08:20.259689 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:20.259693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:20.259708 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:08:20.259789 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:20.259794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:20.259870 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:20.259996 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:20.260725 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:20.261550 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:20.262966 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:20.263385 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:20.263519 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:20.267634 (+ 4115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.195:36099: BlockingRecv error: recv got EOF from 127.5.198.195:36099 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 01:08:20.268059 15147 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 01:08:20 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:20 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:20.436659 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:20.427035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:38363)
0929 01:08:20.427720 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:20.427744 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:20.427887 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:20.428549 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:20.428555 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:20.429177 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:20.429382 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:20.429394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:20.430668 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:20.430672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:20.431668 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:20.431680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:20.432353 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:20.432357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:20.432540 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:20.433164 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:20.433183 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:20.436342 (+ 3159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":422,"threads_started":1}
W20260929 01:08:20.436789 14883 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:20.427108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:38363 (local address 127.5.198.254:37093)
0929 01:08:20.427354 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:20.427363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:20.427891 (+ 528us) server_negotiation.cc:407] Connection header received
0929 01:08:20.428027 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:20.428066 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:20.428180 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:20.428307 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:20.429558 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:20.430529 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:20.431796 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:20.432197 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:20.432289 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:20.436566 (+ 4277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:38363: BlockingRecv error: recv got EOF from 127.5.198.194:38363 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 01:08:21 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:21 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:21.158236 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:21.148868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:49569)
0929 01:08:21.149409 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:21.149426 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:21.149554 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:21.149918 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:21.149921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:21.150330 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:21.150594 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:21.150605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:21.151845 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:21.151849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:21.152944 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:21.152965 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:21.153700 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:21.153706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:21.153837 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:21.154480 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:21.154498 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:21.157998 (+ 3500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":309,"threads_started":1}
W20260929 01:08:21.158459 15159 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:21.148953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:49569 (local address 127.5.198.254:37093)
0929 01:08:21.149386 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:21.149391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:21.149526 (+ 135us) server_negotiation.cc:407] Connection header received
0929 01:08:21.149661 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:21.149665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:21.149747 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:21.149840 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:21.150746 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:21.151653 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:21.153107 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:21.153527 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:21.153629 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:21.158192 (+ 4563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:49569: BlockingRecv error: recv got EOF from 127.5.198.193:49569 (error 108)
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":253,"threads_started":1}
Sep 29 01:08:21 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:21 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:21.278015 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:21.268664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:52433)
0929 01:08:21.269227 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:21.269243 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:21.269407 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:21.269741 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:21.269745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:21.270257 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:21.270405 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:21.270414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:21.271427 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:21.271433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:21.272373 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:21.272384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:21.273135 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:21.273142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:21.273301 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:21.273932 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:21.273955 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:21.277770 (+ 3815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":340,"threads_started":1}
W20260929 01:08:21.278246 15159 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:21.268720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:52433 (local address 127.5.198.254:37093)
0929 01:08:21.269055 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:21.269063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:21.269326 (+ 263us) server_negotiation.cc:407] Connection header received
0929 01:08:21.269524 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:21.269530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:21.269607 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:21.269711 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:21.270543 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:21.271284 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:21.272539 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:21.272957 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:21.273098 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:21.277987 (+ 4889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.195:52433: BlockingRecv error: recv got EOF from 127.5.198.195:52433 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
Sep 29 01:08:21 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:21 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:21.446832 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:21.437569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:60333)
0929 01:08:21.438115 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:21.438137 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:21.438261 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:21.438599 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:21.438604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:21.439014 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:21.439278 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:21.439293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:21.440676 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:21.440679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:21.441614 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:21.441625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:21.442260 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:21.442264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:21.442382 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:21.443008 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:21.443026 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:21.446591 (+ 3565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":303,"threads_started":1}
W20260929 01:08:21.447080 15159 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:21.437655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:60333 (local address 127.5.198.254:37093)
0929 01:08:21.437831 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:21.437836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:21.438228 (+ 392us) server_negotiation.cc:407] Connection header received
0929 01:08:21.438373 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:21.438377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:21.438437 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:21.438524 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:21.439444 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:21.440542 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:21.441739 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:21.442132 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:21.442224 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:21.446777 (+ 4553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:60333: BlockingRecv error: recv got EOF from 127.5.198.194:60333 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 01:08:22 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:22 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:22.168184 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:22.159286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:58165)
0929 01:08:22.159812 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:22.159860 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:22.160049 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:22.160398 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:22.160401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:22.160809 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:22.161025 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:22.161034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:22.162126 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:22.162130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:22.163162 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:22.163175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:22.163833 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:22.163838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:22.163982 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:22.164670 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:22.164688 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:22.167890 (+ 3202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":239,"threads_started":1}
W20260929 01:08:22.168408 15163 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:22.159556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:58165 (local address 127.5.198.254:37093)
0929 01:08:22.160159 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:22.160164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:22.160178 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:08:22.160219 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:22.160222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:22.160275 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:22.160362 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:22.161218 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:22.161984 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:22.163295 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:22.163680 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:22.163776 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:22.168140 (+ 4364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:58165: BlockingRecv error: recv got EOF from 127.5.198.193:58165 (error 108)
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":391,"threads_started":1}
Sep 29 01:08:22 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:22 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:22.287305 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:22.278831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:46537)
0929 01:08:22.279322 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:22.279338 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:22.279462 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:22.279726 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:22.279729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:22.280093 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:22.280266 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:22.280289 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:22.281429 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:22.281433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:22.282339 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:22.282349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:22.282988 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:22.282992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:22.283116 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:22.283766 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:22.283786 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:22.287055 (+ 3269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":298,"threads_started":1}
W20260929 01:08:22.287395 15163 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:22.278950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:46537 (local address 127.5.198.254:37093)
0929 01:08:22.279194 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:22.279200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:22.279411 (+ 211us) server_negotiation.cc:407] Connection header received
0929 01:08:22.279545 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:22.279549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:22.279607 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:22.279684 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:22.280401 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:22.281292 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:22.282472 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:22.282860 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:22.282950 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:22.287221 (+ 4271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.195:46537: BlockingRecv error: recv got EOF from 127.5.198.195:46537 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 01:08:22 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:22 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:22.457615 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:22.447757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:44465)
0929 01:08:22.448422 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:22.448445 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:22.448615 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:22.449033 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:22.449037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:22.449365 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:22.449531 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:22.449540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:22.450637 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:22.450644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:22.451724 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:22.451737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:22.452583 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:22.452588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:22.452721 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:22.453337 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:22.453355 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:22.457262 (+ 3907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":357,"threads_started":1}
W20260929 01:08:22.457813 15163 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:22.447880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:44465 (local address 127.5.198.254:37093)
0929 01:08:22.448076 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:22.448080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:22.448566 (+ 486us) server_negotiation.cc:407] Connection header received
0929 01:08:22.448748 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:22.448752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:22.448836 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:22.448946 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:22.449659 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:22.450368 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:22.451878 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:22.452382 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:22.452560 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:22.457533 (+ 4973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:44465: BlockingRecv error: recv got EOF from 127.5.198.194:44465 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 01:08:23 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:23 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:23.178341 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:23.169306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:47449)
0929 01:08:23.169834 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:23.169854 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:23.169980 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:23.170267 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:23.170270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:23.170656 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:23.170813 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:23.170821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:23.171951 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:23.171958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:23.173243 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:23.173258 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:23.173926 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:23.173933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:23.174131 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:23.174715 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:23.174732 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:23.178091 (+ 3359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":305,"threads_started":1}
W20260929 01:08:23.178560 15167 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:23.169355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:47449 (local address 127.5.198.254:37093)
0929 01:08:23.169859 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:23.169863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:23.169936 (+ 73us) server_negotiation.cc:407] Connection header received
0929 01:08:23.170075 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:23.170079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:23.170140 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:23.170227 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:23.170940 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:23.171760 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:23.173390 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:23.173777 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:23.173912 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:23.178282 (+ 4370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:47449: BlockingRecv error: recv got EOF from 127.5.198.193:47449 (error 108)
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":291,"threads_started":1}
Sep 29 01:08:23 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:23 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:23.297977 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:23.288317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:34009)
0929 01:08:23.288864 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:23.288881 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:23.288990 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:23.289258 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:23.289262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:23.289565 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:23.289732 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:23.289740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:23.290922 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:23.290927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:23.291986 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:23.292002 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:23.293506 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:23.293514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:23.293659 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:23.294353 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:23.294372 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:23.297731 (+ 3359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":284,"threads_started":1}
W20260929 01:08:23.298177 15167 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:23.288353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:34009 (local address 127.5.198.254:37093)
0929 01:08:23.288622 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:23.288630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:23.288957 (+ 327us) server_negotiation.cc:407] Connection header received
0929 01:08:23.289082 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:23.289085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:23.289142 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:23.289222 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:23.289883 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:23.290783 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:23.292253 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:23.293243 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:23.293362 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:23.297987 (+ 4625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.195:34009: BlockingRecv error: recv got EOF from 127.5.198.195:34009 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 29 01:08:23 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:23 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:23.468371 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:23.458910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:47303)
0929 01:08:23.459495 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:23.459516 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:23.459624 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:23.459951 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:23.459954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:23.460318 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:23.460502 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:23.460512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:23.461904 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:23.461910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:23.463060 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:23.463078 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:23.463805 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:23.463809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:23.463925 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:23.464545 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:23.464573 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:23.468092 (+ 3519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":323,"threads_started":1}
W20260929 01:08:23.468513 15167 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:23.458937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:47303 (local address 127.5.198.254:37093)
0929 01:08:23.459177 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:23.459188 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:23.459611 (+ 423us) server_negotiation.cc:407] Connection header received
0929 01:08:23.459736 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:23.459740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:23.459816 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:23.459909 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:23.460634 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:23.461767 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:23.463213 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:23.463675 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:23.463788 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:23.468250 (+ 4462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:47303: BlockingRecv error: recv got EOF from 127.5.198.194:47303 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 01:08:24 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:24 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:24.188644 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:24.179310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:60787)
0929 01:08:24.179867 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:24.179884 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:24.180008 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:24.180364 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:24.180368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:24.180820 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:24.181090 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:24.181106 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:24.182346 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:24.182352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:24.183574 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:24.183587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:24.184223 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:24.184228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:24.184349 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:24.185078 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:24.185098 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:24.188346 (+ 3248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":314,"threads_started":1}
W20260929 01:08:24.188827 15171 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:24.179350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:60787 (local address 127.5.198.254:37093)
0929 01:08:24.179838 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:24.179844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:24.179980 (+ 136us) server_negotiation.cc:407] Connection header received
0929 01:08:24.180127 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:24.180134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:24.180210 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:24.180309 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:24.181285 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:24.182199 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:24.183704 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:24.184096 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:24.184190 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:24.188604 (+ 4414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:60787: BlockingRecv error: recv got EOF from 127.5.198.193:60787 (error 108)
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":698,"threads_started":1}
Sep 29 01:08:24 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:24 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:24.308156 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:24.299193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:34227)
0929 01:08:24.299690 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:24.299706 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:24.299808 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:24.300099 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:24.300102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:24.300552 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:24.300757 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:24.300766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:24.302101 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:24.302107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:24.303147 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:24.303159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:24.303767 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:24.303771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:24.303906 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:24.304446 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:24.304514 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:24.307914 (+ 3400us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":271,"threads_started":1}
W20260929 01:08:24.308696 15171 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:24.299310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:34227 (local address 127.5.198.254:37093)
0929 01:08:24.299494 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:24.299499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:24.299781 (+ 282us) server_negotiation.cc:407] Connection header received
0929 01:08:24.299919 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:24.299922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:24.299979 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:24.300062 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:24.300905 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:24.301953 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:24.303273 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:24.303651 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:24.303741 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:24.308517 (+ 4776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.195:34227: BlockingRecv error: recv got EOF from 127.5.198.195:34227 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 01:08:24 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:24 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:24.478385 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:24.469316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:44445)
0929 01:08:24.470022 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:24.470051 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:24.470209 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:24.470559 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:24.470564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:24.470990 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:24.471132 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:24.471140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:24.472102 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:24.472110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:24.473084 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:24.473095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:24.473823 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:24.473829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:24.474026 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:24.474668 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:24.474685 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:24.478150 (+ 3465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":247,"threads_started":1}
W20260929 01:08:24.478711 15171 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:24.469533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:44445 (local address 127.5.198.254:37093)
0929 01:08:24.469729 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:24.469735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:24.470142 (+ 407us) server_negotiation.cc:407] Connection header received
0929 01:08:24.470335 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:24.470342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:24.470422 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:24.470528 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:24.471274 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:24.471973 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:24.473224 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:24.473676 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:24.473812 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:24.478406 (+ 4594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:44445: BlockingRecv error: recv got EOF from 127.5.198.194:44445 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 01:08:25 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:25 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:25.205204 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:25.189638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:55253)
0929 01:08:25.190110 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:25.190127 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:25.190230 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:25.190733 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:25.190736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:25.191120 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:25.191279 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:25.191287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:25.192453 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:25.192457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:25.193565 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:25.193577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:25.194252 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:25.194258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:25.194441 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:25.201295 (+ 6854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:25.201326 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:25.204839 (+ 3513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":261,"threads_started":1}
W20260929 01:08:25.205456 15175 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:25.189703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:55253 (local address 127.5.198.254:37093)
0929 01:08:25.190463 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:25.190467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:25.190483 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:08:25.190528 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:25.190532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:25.190602 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:25.190687 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:25.191418 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:25.192290 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:25.193699 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:25.194099 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:25.194233 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:25.205037 (+ 10804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:55253: BlockingRecv error: recv got EOF from 127.5.198.193:55253 (error 108)
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":545,"threads_started":1}
Sep 29 01:08:25 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:25 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:25.317885 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:25.309046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:48233)
0929 01:08:25.309548 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:25.309565 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:25.309671 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:25.309975 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:25.309979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:25.310368 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:25.310502 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:25.310510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:25.311637 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:25.311643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:25.312910 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:25.312923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:25.313556 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:25.313561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:25.313673 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:25.314251 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:25.314271 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:25.317657 (+ 3386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":260,"threads_started":1}
W20260929 01:08:25.317993 15175 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:25.309115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:48233 (local address 127.5.198.254:37093)
0929 01:08:25.309268 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:25.309272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:25.309635 (+ 363us) server_negotiation.cc:407] Connection header received
0929 01:08:25.309763 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:25.309766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:25.309825 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:25.309936 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:25.310622 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:25.311496 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:25.313034 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:25.313432 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:25.313524 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:25.317836 (+ 4312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.195:48233: BlockingRecv error: recv got EOF from 127.5.198.195:48233 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 01:08:25 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:25 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:25.487656 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:25.479234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:48807)
0929 01:08:25.479729 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:25.479750 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:25.479860 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:25.480175 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:25.480178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:25.480588 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:25.480750 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:25.480758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:25.481778 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:25.481781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:25.482709 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:25.482720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:25.483394 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:25.483398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:25.483516 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:25.484129 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:25.484146 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:25.487445 (+ 3299us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":281,"threads_started":1}
W20260929 01:08:25.487931 15175 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:25.479325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:48807 (local address 127.5.198.254:37093)
0929 01:08:25.479509 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:25.479513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:25.479820 (+ 307us) server_negotiation.cc:407] Connection header received
0929 01:08:25.479979 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:25.479983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:25.480053 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:25.480133 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:25.480911 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:25.481657 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:25.482878 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:25.483264 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:25.483358 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:25.487624 (+ 4266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:48807: BlockingRecv error: recv got EOF from 127.5.198.194:48807 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 01:08:26 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:26 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:26.215790 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:26.206109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:38581)
0929 01:08:26.206918 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:26.206934 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:26.207048 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:26.207438 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:26.207441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:26.207819 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:26.208098 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:26.208111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:26.209433 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:26.209446 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:26.210614 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:26.210632 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:26.211352 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:26.211357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:26.211496 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:26.212125 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:26.212142 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:26.215560 (+ 3418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":478,"threads_started":1}
W20260929 01:08:26.216090 15179 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:26.206199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:38581 (local address 127.5.198.254:37093)
0929 01:08:26.206663 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:26.206668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:26.207016 (+ 348us) server_negotiation.cc:407] Connection header received
0929 01:08:26.207173 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:26.207177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:26.207264 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:26.207357 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:26.208269 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:26.209265 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:26.210752 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:26.211182 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:26.211287 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:26.215774 (+ 4487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:38581: BlockingRecv error: recv got EOF from 127.5.198.193:38581 (error 108)
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":247,"threads_started":1}
Sep 29 01:08:26 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:26 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:26.327824 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:26.318938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:38009)
0929 01:08:26.319488 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:26.319506 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:26.319625 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:26.319909 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:26.319912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:26.320281 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:26.320442 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:26.320450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:26.321536 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:26.321542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:26.322454 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:26.322466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:26.323192 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:26.323197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:26.323381 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:26.323972 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:26.323991 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:26.327525 (+ 3534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":268,"threads_started":1}
W20260929 01:08:26.327961 15179 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:26.319029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:38009 (local address 127.5.198.254:37093)
0929 01:08:26.319236 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:26.319242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:26.319641 (+ 399us) server_negotiation.cc:407] Connection header received
0929 01:08:26.319718 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:26.319721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:26.319820 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:26.319897 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:26.320640 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:26.321396 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:26.322606 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:26.323038 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:26.323176 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:26.327683 (+ 4507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.195:38009: BlockingRecv error: recv got EOF from 127.5.198.195:38009 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 01:08:26 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:26 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:26.497740 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:26.488651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:37767)
0929 01:08:26.489247 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:26.489266 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:26.489427 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:26.489767 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:26.489772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:26.490196 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:26.490362 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:26.490373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:26.491362 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:26.491371 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:26.492340 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:26.492352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:26.493064 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:26.493070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:26.493244 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:26.493886 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:26.493904 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:26.497501 (+ 3597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":306,"threads_started":1}
W20260929 01:08:26.498149 15179 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:26.488681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:37767 (local address 127.5.198.254:37093)
0929 01:08:26.488951 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:26.488959 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:26.489367 (+ 408us) server_negotiation.cc:407] Connection header received
0929 01:08:26.489550 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:26.489555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:26.489632 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:26.489734 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:26.490516 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:26.491228 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:26.492505 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:26.492906 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:26.493036 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:26.497710 (+ 4674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:37767: BlockingRecv error: recv got EOF from 127.5.198.194:37767 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Sep 29 01:08:27 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:27 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:27.226495 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:27.216998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:50843)
0929 01:08:27.217839 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:27.217879 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:27.218020 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:27.218369 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:27.218394 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:27.218932 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:27.219138 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:27.219147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:27.220195 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:27.220199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:27.221158 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:27.221168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:27.221940 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:27.221946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:27.222087 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:27.222689 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:27.222707 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:27.226221 (+ 3514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":562,"threads_started":1}
W20260929 01:08:27.226709 15183 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:27.217045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:50843 (local address 127.5.198.254:37093)
0929 01:08:27.217659 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:27.217666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:27.218007 (+ 341us) server_negotiation.cc:407] Connection header received
0929 01:08:27.218130 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:27.218133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:27.218191 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:27.218334 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:27.219291 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:27.220047 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:27.221293 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:27.221736 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:27.221883 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:27.226469 (+ 4586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:50843: BlockingRecv error: recv got EOF from 127.5.198.193:50843 (error 108)
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":328,"threads_started":1}
Sep 29 01:08:27 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:27 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:27.337814 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:27.328732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:38443)
0929 01:08:27.329333 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:27.329349 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:27.329463 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:27.329803 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:27.329806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:27.330248 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:27.330424 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:27.330433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:27.331439 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:27.331443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:27.332426 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:27.332437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:27.333161 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:27.333165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:27.333297 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:27.333842 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:27.333861 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:27.337465 (+ 3604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":303,"threads_started":1}
W20260929 01:08:27.338157 15183 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:27.328888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:38443 (local address 127.5.198.254:37093)
0929 01:08:27.329123 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:27.329132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:27.329435 (+ 303us) server_negotiation.cc:407] Connection header received
0929 01:08:27.329560 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:27.329564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:27.329621 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:27.329701 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:27.330564 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:27.331296 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:27.332608 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:27.333011 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:27.333106 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:27.337654 (+ 4548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.195:38443: BlockingRecv error: recv got EOF from 127.5.198.195:38443 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 01:08:27 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:27 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:27.509405 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:27.498700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:46515)
0929 01:08:27.499314 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:27.499347 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:27.499521 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:27.499897 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:27.499902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:27.500368 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:27.500564 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:27.500573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:27.501883 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:27.501889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:27.503044 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:27.503057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:27.503842 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:27.503848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:27.504036 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:27.504911 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:27.504945 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:27.509019 (+ 4074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":314,"threads_started":1}
W20260929 01:08:27.509591 15183 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:27.498864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:46515 (local address 127.5.198.254:37093)
0929 01:08:27.499111 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:27.499120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:27.499465 (+ 345us) server_negotiation.cc:407] Connection header received
0929 01:08:27.499670 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:27.499678 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:27.499747 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:27.499861 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:27.500719 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:27.501727 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:27.503202 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:27.503651 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:27.503807 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:27.509265 (+ 5458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:46515: BlockingRecv error: recv got EOF from 127.5.198.194:46515 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 01:08:28 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:28 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:28.238904 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:28.227832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:54747)
0929 01:08:28.228445 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:28.228489 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:28.228613 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:28.228945 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:28.228949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:28.229355 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:28.229510 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:28.229519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:28.231159 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:28.231164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:28.233100 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:28.233120 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:28.233923 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:28.233929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:28.234101 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:28.234742 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:28.234759 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:28.238506 (+ 3747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":253,"threads_started":1}
W20260929 01:08:28.239039 15187 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:28.227904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:54747 (local address 127.5.198.254:37093)
0929 01:08:28.228628 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:28.228634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:28.228653 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:08:28.228710 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:28.228715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:28.228803 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:28.228908 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:28.229659 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:28.231015 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:28.233262 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:28.233770 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:28.233906 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:28.238712 (+ 4806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:54747: BlockingRecv error: recv got EOF from 127.5.198.193:54747 (error 108)
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":477,"threads_started":1}
Sep 29 01:08:28 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:28 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:28.348021 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:28.338775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:60613)
0929 01:08:28.339341 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:28.339362 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:28.339500 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:28.339871 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:28.339877 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:28.340367 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:28.340610 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:28.340620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:28.341650 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:28.341653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:28.342583 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:28.342595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:28.343256 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:28.343261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:28.343385 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:28.343956 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:28.343974 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:28.347659 (+ 3685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":301,"threads_started":1}
W20260929 01:08:28.348129 15187 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:28.338829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:60613 (local address 127.5.198.254:37093)
0929 01:08:28.339040 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:28.339047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:28.339451 (+ 404us) server_negotiation.cc:407] Connection header received
0929 01:08:28.339630 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:28.339637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:28.339716 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:28.339826 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:28.340752 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:28.341516 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:28.342715 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:28.343100 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:28.343197 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:28.347892 (+ 4695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.195:60613: BlockingRecv error: recv got EOF from 127.5.198.195:60613 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 01:08:28 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:28 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:28.519872 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:28.510424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:57317)
0929 01:08:28.510954 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:28.510976 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:28.511082 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:28.511479 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:28.511483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:28.511902 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:28.512041 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:28.512049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:28.513167 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:28.513171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:28.514133 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:28.514145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:28.514894 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:28.514898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:28.515029 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:28.515626 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:28.515644 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:28.519549 (+ 3905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":285,"threads_started":1}
W20260929 01:08:28.519985 15187 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:28.510473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:57317 (local address 127.5.198.254:37093)
0929 01:08:28.510631 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:28.510636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:28.511056 (+ 420us) server_negotiation.cc:407] Connection header received
0929 01:08:28.511223 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:28.511228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:28.511307 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:28.511396 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:28.512191 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:28.513036 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:28.514275 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:28.514683 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:28.514790 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:28.519719 (+ 4929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:57317: BlockingRecv error: recv got EOF from 127.5.198.194:57317 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 01:08:29 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:29 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:29.249002 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:29.239894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:43275)
0929 01:08:29.240505 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:29.240530 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:29.240640 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:29.240951 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:29.240954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:29.241278 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:29.241463 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:29.241472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:29.242446 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:29.242450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:29.243433 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:29.243445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:29.244153 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:29.244158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:29.244276 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:29.244999 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:29.245016 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:29.248672 (+ 3656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":249,"threads_started":1}
W20260929 01:08:29.249236 15191 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:29.240092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:43275 (local address 127.5.198.254:37093)
0929 01:08:29.240633 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:29.240637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:29.240653 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:08:29.240730 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:29.240733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:29.240805 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:29.240923 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:29.241582 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:29.242303 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:29.243561 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:29.244000 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:29.244093 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:29.248935 (+ 4842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:43275: BlockingRecv error: recv got EOF from 127.5.198.193:43275 (error 108)
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":247,"threads_started":1}
Sep 29 01:08:29 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:29 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:29.358359 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:29.348994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:54527)
0929 01:08:29.349519 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:29.349535 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:29.349683 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:29.350165 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:29.350168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:29.350537 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:29.350675 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:29.350682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:29.351805 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:29.351811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:29.352872 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:29.352883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:29.353529 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:29.353534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:29.353679 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:29.354244 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:29.354261 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:29.358017 (+ 3756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":327,"threads_started":1}
W20260929 01:08:29.358430 15191 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:29.349276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:54527 (local address 127.5.198.254:37093)
0929 01:08:29.349474 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:29.349482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:29.349608 (+ 126us) server_negotiation.cc:407] Connection header received
0929 01:08:29.349927 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:29.349939 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:29.350036 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:29.350120 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:29.350792 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:29.351669 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:29.352994 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:29.353370 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:29.353471 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:29.358181 (+ 4710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.195:54527: BlockingRecv error: recv got EOF from 127.5.198.195:54527 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 01:08:29 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:29 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:29.529948 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:29.520757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:39847)
0929 01:08:29.521467 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:29.521494 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:29.521652 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:29.521990 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:29.521994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:29.522383 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:29.522539 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:29.522548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:29.523592 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:29.523598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:29.524598 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:29.524609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:29.525271 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:29.525276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:29.525448 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:29.526076 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:29.526103 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:29.529572 (+ 3469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":454,"threads_started":1}
W20260929 01:08:29.529974 15191 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:29.520834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:39847 (local address 127.5.198.254:37093)
0929 01:08:29.521075 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:29.521080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:29.521579 (+ 499us) server_negotiation.cc:407] Connection header received
0929 01:08:29.521772 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:29.521780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:29.521857 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:29.521957 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:29.522694 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:29.523442 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:29.524742 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:29.525128 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:29.525263 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:29.529744 (+ 4481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:39847: BlockingRecv error: recv got EOF from 127.5.198.194:39847 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 29 01:08:30 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:30 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:30.266850 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:30.250073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:54215)
0929 01:08:30.250661 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:30.250679 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:30.250795 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:30.251181 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:30.251185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:30.251646 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:30.251841 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:30.251850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:30.252970 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:30.252975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:30.254226 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:30.254237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:30.255084 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:30.255088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:30.255209 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:30.262398 (+ 7189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:30.262417 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:30.266490 (+ 4073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":339,"threads_started":1}
W20260929 01:08:30.267210 15195 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:30.250344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:54215 (local address 127.5.198.254:37093)
0929 01:08:30.250864 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:30.250868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:30.250882 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:08:30.250927 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:30.250931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:30.250985 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:30.251112 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:30.251985 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:30.252815 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:30.254375 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:30.254959 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:30.255051 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:30.266707 (+ 11656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:54215: BlockingRecv error: recv got EOF from 127.5.198.193:54215 (error 108)
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":287,"threads_started":1}
Sep 29 01:08:30 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:30 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:30.369033 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:30.359307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:36241)
0929 01:08:30.359869 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:30.359897 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:30.360041 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:30.360367 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:30.360370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:30.360866 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:30.361022 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:30.361030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:30.362478 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:30.362485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:30.363784 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:30.363797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:30.364524 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:30.364531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:30.364659 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:30.365216 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:30.365235 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:30.368694 (+ 3459us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":290,"threads_started":1}
W20260929 01:08:30.369086 15195 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:30.359413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:36241 (local address 127.5.198.254:37093)
0929 01:08:30.359575 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:30.359579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:30.359986 (+ 407us) server_negotiation.cc:407] Connection header received
0929 01:08:30.360139 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:30.360143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:30.360239 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:30.360333 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:30.361168 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:30.362280 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:30.363945 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:30.364338 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:30.364439 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:30.368852 (+ 4413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.195:36241: BlockingRecv error: recv got EOF from 127.5.198.195:36241 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 01:08:30 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:30 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:30.539754 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:30.530945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:57007)
0929 01:08:30.531409 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:30.531429 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:30.531541 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:30.531903 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:30.531906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:30.532317 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:30.532525 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:30.532533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:30.533515 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:30.533519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:30.534478 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:30.534489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:30.535178 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:30.535182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:30.535313 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:30.535935 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:30.535953 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:30.539436 (+ 3483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":281,"threads_started":1}
W20260929 01:08:30.539875 15195 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:30.531008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:57007 (local address 127.5.198.254:37093)
0929 01:08:30.531233 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:30.531243 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:30.531525 (+ 282us) server_negotiation.cc:407] Connection header received
0929 01:08:30.531667 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:30.531671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:30.531744 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:30.531863 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:30.532649 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:30.533396 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:30.534615 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:30.535038 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:30.535140 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:30.539622 (+ 4482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:57007: BlockingRecv error: recv got EOF from 127.5.198.194:57007 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 01:08:31 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:31 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:31.277004 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:31.267852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:38223)
0929 01:08:31.268399 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:31.268422 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:31.268614 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:31.269139 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:31.269144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:31.269696 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:31.269864 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:31.269873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:31.270919 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:31.270924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:31.271835 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:31.271846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:31.272547 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:31.272551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:31.272691 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:31.273370 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:31.273387 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:31.276730 (+ 3343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":255,"threads_started":1}
W20260929 01:08:31.277191 15199 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:31.267935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:38223 (local address 127.5.198.254:37093)
0929 01:08:31.268564 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:31.268570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:31.268589 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:08:31.268783 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:31.268787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:31.268918 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:31.269040 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:31.270034 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:31.270767 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:31.271980 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:31.272372 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:31.272523 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:31.276974 (+ 4451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:38223: BlockingRecv error: recv got EOF from 127.5.198.193:38223 (error 108)
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":326,"threads_started":1}
Sep 29 01:08:31 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:31 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:31.379035 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:31.370030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:50869)
0929 01:08:31.370553 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:31.370569 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:31.370675 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:31.371077 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:31.371081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:31.371495 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:31.371681 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:31.371689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:31.372830 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:31.372834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:31.373772 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:31.373783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:31.374492 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:31.374497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:31.374608 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:31.375274 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:31.375294 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:31.378827 (+ 3533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":293,"threads_started":1}
W20260929 01:08:31.379330 15199 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:31.370233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:50869 (local address 127.5.198.254:37093)
0929 01:08:31.370413 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:31.370417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:31.370641 (+ 224us) server_negotiation.cc:407] Connection header received
0929 01:08:31.370849 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:31.370853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:31.370914 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:31.371002 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:31.371837 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:31.372675 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:31.373969 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:31.374356 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:31.374467 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:31.379020 (+ 4553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.195:50869: BlockingRecv error: recv got EOF from 127.5.198.195:50869 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 01:08:31 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:31 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:31.550715 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:31.540885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:33401)
0929 01:08:31.541442 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:31.541461 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:31.541570 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:31.541986 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:31.541989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:31.542473 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:31.542684 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:31.542700 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:31.544001 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:31.544006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:31.545235 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:31.545247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:31.545926 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:31.545930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:31.546062 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:31.546718 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:31.546735 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:31.550347 (+ 3612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":308,"threads_started":1}
W20260929 01:08:31.550782 15199 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:31.540884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:33401 (local address 127.5.198.254:37093)
0929 01:08:31.541055 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:31.541060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:31.541543 (+ 483us) server_negotiation.cc:407] Connection header received
0929 01:08:31.541708 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:31.541713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:31.541833 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:31.541929 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:31.542861 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:31.543841 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:31.545340 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:31.545751 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:31.545854 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:31.550545 (+ 4691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:33401: BlockingRecv error: recv got EOF from 127.5.198.194:33401 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 01:08:32 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:32 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:32.287547 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:32.278001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:44479)
0929 01:08:32.278782 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:32.278798 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:32.278953 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:32.279386 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:32.279392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:32.279849 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:32.280005 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:32.280014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:32.281058 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:32.281063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:32.282006 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:32.282019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:32.282706 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:32.282711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:32.282863 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:32.283442 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:32.283460 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:32.287300 (+ 3840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":555,"threads_started":1}
W20260929 01:08:32.287760 15203 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:32.278089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:44479 (local address 127.5.198.254:37093)
0929 01:08:32.278648 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:32.278654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:32.278889 (+ 235us) server_negotiation.cc:407] Connection header received
0929 01:08:32.279072 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:32.279076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:32.279168 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:32.279269 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:32.280160 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:32.280896 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:32.282152 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:32.282556 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:32.282649 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:32.287489 (+ 4840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:44479: BlockingRecv error: recv got EOF from 127.5.198.193:44479 (error 108)
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":279,"threads_started":1}
Sep 29 01:08:32 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:32 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:32.388887 15204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:32.379936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:39929)
0929 01:08:32.380445 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:32.380482 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:32.380587 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:32.380921 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:32.380924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:32.381310 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:32.381470 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:32.381478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:32.382623 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:32.382626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:32.383513 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:32.383523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:32.384320 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:32.384324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:32.384442 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:32.385116 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:32.385134 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:32.388648 (+ 3514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":282,"threads_started":1}
W20260929 01:08:32.388978 15203 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:32.380069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:39929 (local address 127.5.198.254:37093)
0929 01:08:32.380340 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:32.380347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:32.380560 (+ 213us) server_negotiation.cc:407] Connection header received
0929 01:08:32.380688 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:32.380692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:32.380755 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:32.380852 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:32.381630 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:32.382479 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:32.383653 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:32.384160 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:32.384285 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:32.388799 (+ 4514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.195:39929: BlockingRecv error: recv got EOF from 127.5.198.195:39929 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 01:08:32 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:32 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:32.560926 15205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:32.551696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:58345)
0929 01:08:32.552169 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:32.552190 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:32.552297 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:32.552640 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:32.552644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:32.553084 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:32.553231 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:32.553240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:32.554225 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:32.554228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:32.555152 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:32.555162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:32.555844 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:32.555848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:32.556082 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:32.556763 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:32.556780 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:32.560614 (+ 3834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":272,"threads_started":1}
W20260929 01:08:32.561138 15203 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:32.551836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:58345 (local address 127.5.198.254:37093)
0929 01:08:32.552074 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:32.552080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:32.552263 (+ 183us) server_negotiation.cc:407] Connection header received
0929 01:08:32.552394 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:32.552398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:32.552489 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:32.552568 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:32.553378 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:32.554090 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:32.555289 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:32.555692 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:32.555792 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:32.560806 (+ 5014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:58345: BlockingRecv error: recv got EOF from 127.5.198.194:58345 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 29 01:08:33 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:33 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.193@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:33.298729 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:33.288908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.193:59501)
0929 01:08:33.289413 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:33.289429 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:33.289523 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:33.289856 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:33.289859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:33.290198 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:33.290372 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:33.290380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:33.291378 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:33.291382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:33.292688 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:33.292705 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:33.293709 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:33.293715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:33.293899 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:33.294567 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:33.294591 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:33.298346 (+ 3755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":264,"threads_started":1}
W20260929 01:08:33.298766 15207 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:33.289020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:59501 (local address 127.5.198.254:37093)
0929 01:08:33.289487 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:33.289491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:33.289507 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:08:33.289626 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:33.289629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:33.289692 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:33.289783 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:33.290509 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:33.291237 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:33.292889 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:33.293542 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:33.293675 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:33.298525 (+ 4850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.193:59501: BlockingRecv error: recv got EOF from 127.5.198.193:59501 (error 108)
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":253,"threads_started":1}
Sep 29 01:08:33 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:33 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.195@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:33.399003 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:33.389966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.195:55399)
0929 01:08:33.390431 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:33.390457 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:33.390588 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:33.390975 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:33.390979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:33.391461 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:33.391653 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:33.391685 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:33.392831 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:33.392837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:33.393722 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:33.393732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:33.394387 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:33.394391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:33.394510 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:33.395090 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:33.395112 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:33.398742 (+ 3630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":253,"threads_started":1}
W20260929 01:08:33.399070 15207 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:33.390051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:55399 (local address 127.5.198.254:37093)
0929 01:08:33.390279 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:33.390287 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:33.390538 (+ 251us) server_negotiation.cc:407] Connection header received
0929 01:08:33.390703 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:33.390706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:33.390807 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:33.390939 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:33.391823 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:33.392710 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:33.393864 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:33.394266 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:33.394368 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:33.398898 (+ 4530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.195:55399: BlockingRecv error: recv got EOF from 127.5.198.195:55399 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 29 01:08:33 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:08:33 dist-test-slave-41cl krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.198.194@KRBTEST.COM for oryx/127.5.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:08:33.571363 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:33.562406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37093 (local address 127.5.198.194:58627)
0929 01:08:33.563073 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:33.563099 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:33.563255 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:33.563618 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:33.563622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:33.564014 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:33.564217 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:33.564231 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:33.565285 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:33.565291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:33.566207 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:33.566217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:33.566921 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:33.566927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:33.567100 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:33.567649 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:33.567666 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:33.571122 (+ 3456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.198.254:37093: Server oryx/127.5.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":371,"threads_started":1}
W20260929 01:08:33.571539 15207 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:33.562442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:58627 (local address 127.5.198.254:37093)
0929 01:08:33.562620 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:33.562625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:33.563186 (+ 561us) server_negotiation.cc:407] Connection header received
0929 01:08:33.563369 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:33.563395 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:33.563477 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:33.563582 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:33.564361 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:33.565149 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:33.566342 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:33.566752 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:33.566882 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:33.571319 (+ 4437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.198.194:58627: BlockingRecv error: recv got EOF from 127.5.198.194:58627 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:08:34.096818 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 14733
I20260929 01:08:34.109443 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:34.097758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.5.198.193:43151)
0929 01:08:34.098303 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:34.098311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:34.098338 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:08:34.098399 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:34.098406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:34.098628 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:34.098798 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:34.100144 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:34.101464 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:34.102643 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:34.103156 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:34.103251 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:34.105832 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:34.105840 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:34.105854 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:34.105893 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:34.108945 (+ 3052us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.198.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 01:08:34.109218 (+ 273us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50756: Request ticket server kudu/127.5.198.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":271,"threads_started":1}
W20260929 01:08:34.109570 15211 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50756: Request ticket server kudu/127.5.198.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 01:08:34.109656 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 14733: Not authorized: Client connection negotiation failed: client connection to 127.5.198.193:43151: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.198.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 01:08:34.109872 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 14733
I20260929 01:08:34.127439 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 14885
I20260929 01:08:34.140008 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:34.128451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56938 (local address 127.5.198.194:43525)
0929 01:08:34.129043 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:34.129050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:34.129073 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:08:34.129135 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:34.129142 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:34.129376 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:34.129530 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:34.131290 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:34.132261 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:34.133589 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:34.134121 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:34.134223 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:34.136735 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:34.136755 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:34.136770 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:34.136882 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:34.139475 (+ 2593us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.198.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 01:08:34.139727 (+ 252us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56938: Request ticket server kudu/127.5.198.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":289,"threads_started":1}
W20260929 01:08:34.140112 15212 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56938: Request ticket server kudu/127.5.198.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 01:08:34.140106 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 14885: Not authorized: Client connection negotiation failed: client connection to 127.5.198.194:43525: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.198.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 01:08:34.140251 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 14885
I20260929 01:08:34.157065 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 15018
I20260929 01:08:34.169273 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:34.158030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38046 (local address 127.5.198.195:41991)
0929 01:08:34.158619 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:34.158626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:34.158651 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:08:34.158713 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:34.158720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:34.158987 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:34.159151 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:34.160980 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:34.162008 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:34.163171 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:34.163687 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:34.163785 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:34.166197 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:34.166205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:34.166217 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:34.166255 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:34.168762 (+ 2507us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.198.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 01:08:34.169048 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38046: Request ticket server kudu/127.5.198.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":295,"threads_started":1}
W20260929 01:08:34.169392 15213 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38046: Request ticket server kudu/127.5.198.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 01:08:34.169407 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 15018: Not authorized: Client connection negotiation failed: client connection to 127.5.198.195:41991: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.198.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 01:08:34.169559 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 15018
I20260929 01:08:34.185698 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 14664
I20260929 01:08:34.186932 14726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:34.300062 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 14664
2026-09-29T01:08:34Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21409 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T01:08:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:08:34Z Disabled control of system clock
I20260929 01:08:34.374531 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:44003
--webserver_interface=127.5.198.254
--webserver_port=0
--builtin_ntp_servers=127.5.198.212:45625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:44003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 01:08:34.687801 15218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:34.688081 15218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:34.688232 15218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:34.695428 15218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:08:34.695521 15218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:34.695602 15218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:34.695698 15218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:08:34.695761 15218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:08:34.704859 15218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:45625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:44003
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:44003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:34.706108 15218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:34.707803 15218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:34.719362 15226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:34.719357 15224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:34.719384 15223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:34.720016 15218 server_base.cc:1034] running on GCE node
I20260929 01:08:34.720842 15218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:34.722129 15218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:34.723361 15218 hybrid_clock.cc:648] HybridClock initialized: now 1790644114723326 us; error 42 us; skew 500 ppm
I20260929 01:08:34.723701 15218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:08:34.725822 15218 webserver.cc:460] Webserver started at http://127.5.198.254:42807/ using document root <none> and password file <none>
I20260929 01:08:34.726482 15218 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:34.726601 15218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:34.726913 15218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:34.729673 15218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "70f28b0323434d5d86d58f95ef26540d"
format_stamp: "Formatted at 2026-09-29 01:08:34 on dist-test-slave-41cl"
server_key: "b6c3eadef7d869c76ac226d056e309d2"
server_key_iv: "6ec2dc7dba422ed0e1bc4cb5a5883654"
server_key_version: "encryptionkey@0"
I20260929 01:08:34.730427 15218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "70f28b0323434d5d86d58f95ef26540d"
format_stamp: "Formatted at 2026-09-29 01:08:34 on dist-test-slave-41cl"
server_key: "b6c3eadef7d869c76ac226d056e309d2"
server_key_iv: "6ec2dc7dba422ed0e1bc4cb5a5883654"
server_key_version: "encryptionkey@0"
I20260929 01:08:34.735146 15218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:08:34.738420 15232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:34.739681 15218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:08:34.739894 15218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "70f28b0323434d5d86d58f95ef26540d"
format_stamp: "Formatted at 2026-09-29 01:08:34 on dist-test-slave-41cl"
server_key: "b6c3eadef7d869c76ac226d056e309d2"
server_key_iv: "6ec2dc7dba422ed0e1bc4cb5a5883654"
server_key_version: "encryptionkey@0"
I20260929 01:08:34.740097 15218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:34.770874 15218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:34.771657 15218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:34.771941 15218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:34.790078 15218 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:44003
I20260929 01:08:34.790107 15283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:44003 every 8 connection(s)
I20260929 01:08:34.791599 15218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:08:34.793121 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 15218
I20260929 01:08:34.793300 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:08:34.793952 5915 external_mini_cluster.cc:1428] Setting key 9ce9c0f4ddf243ed40e80cfa7cc923f8
I20260929 01:08:34.795609 15284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:34.808737 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:34.795857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60284 (local address 127.5.198.254:44003)
0929 01:08:34.796895 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:34.796909 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:34.796958 (+ 49us) server_negotiation.cc:407] Connection header received
0929 01:08:34.798836 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:34.798862 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:34.799441 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:34.799858 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:34.800343 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:34.802330 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:34.803718 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:34.804714 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:34.804887 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:34.805987 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:34.805997 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:34.806018 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:08:34.806751 (+ 733us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:34.806841 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:08:34.806866 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:34.806972 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:34.807206 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 01:08:34.807527 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":309,"threads_started":1}
I20260929 01:08:34.809542 15284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d: Bootstrap starting.
I20260929 01:08:34.812629 15284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:34.813562 15284 log.cc:824] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:08:34.816092 15284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d: No bootstrap required, opened a new log
I20260929 01:08:34.823385 15284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f28b0323434d5d86d58f95ef26540d" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44003 } }
I20260929 01:08:34.823776 15284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:34.823905 15284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f28b0323434d5d86d58f95ef26540d, State: Initialized, Role: FOLLOWER
I20260929 01:08:34.824542 15284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f28b0323434d5d86d58f95ef26540d" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44003 } }
I20260929 01:08:34.824762 15284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:08:34.824893 15284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:08:34.825068 15284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:34.827049 15284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f28b0323434d5d86d58f95ef26540d" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44003 } }
I20260929 01:08:34.827522 15284 leader_election.cc:302] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70f28b0323434d5d86d58f95ef26540d; no voters:
I20260929 01:08:34.828063 15284 leader_election.cc:288] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:08:34.828197 15289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:08:34.828917 15289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [term 1 LEADER]: Becoming Leader. State: Replica: 70f28b0323434d5d86d58f95ef26540d, State: Running, Role: LEADER
I20260929 01:08:34.829411 15289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f28b0323434d5d86d58f95ef26540d" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44003 } }
I20260929 01:08:34.830029 15284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:08:34.831862 15291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70f28b0323434d5d86d58f95ef26540d. Latest consensus state: current_term: 1 leader_uuid: "70f28b0323434d5d86d58f95ef26540d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f28b0323434d5d86d58f95ef26540d" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44003 } } }
I20260929 01:08:34.832219 15291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:34.833886 15294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:08:34.833989 15290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70f28b0323434d5d86d58f95ef26540d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f28b0323434d5d86d58f95ef26540d" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44003 } } }
I20260929 01:08:34.834349 15290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:34.839943 15294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:08:34.848308 15294 catalog_manager.cc:1347] Generated new cluster ID: 02ce9f468f7a49f89f69fa559ee62206
I20260929 01:08:34.848429 15294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:08:34.862108 15294 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:08:34.863948 15294 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:08:34.874269 15294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70f28b0323434d5d86d58f95ef26540d: Generated new TSK 0
I20260929 01:08:34.875277 15294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:08:34.883101 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44003
--builtin_ntp_servers=127.5.198.212:45625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 01:08:35.217679 15309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:35.217973 15309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:35.218122 15309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:35.225507 15309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:35.225601 15309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:35.225811 15309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:08:35.235328 15309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:45625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:35.236671 15309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:35.238590 15309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:35.251879 15315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:35.251979 15314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:35.251910 15317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:35.252902 15309 server_base.cc:1034] running on GCE node
I20260929 01:08:35.253592 15309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:35.254565 15309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:35.255788 15309 hybrid_clock.cc:648] HybridClock initialized: now 1790644115255737 us; error 65 us; skew 500 ppm
I20260929 01:08:35.256124 15309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:08:35.258323 15309 webserver.cc:460] Webserver started at http://127.5.198.193:45003/ using document root <none> and password file <none>
I20260929 01:08:35.258991 15309 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:35.259114 15309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:35.259415 15309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:35.262181 15309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "0a65e545f7cb4a1a92b0323f80f07933"
format_stamp: "Formatted at 2026-09-29 01:08:35 on dist-test-slave-41cl"
server_key: "e04c2435d74650a231ec21d70572987c"
server_key_iv: "d28e5677bfd7e2ab160c9a161c54a497"
server_key_version: "encryptionkey@0"
I20260929 01:08:35.262964 15309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "0a65e545f7cb4a1a92b0323f80f07933"
format_stamp: "Formatted at 2026-09-29 01:08:35 on dist-test-slave-41cl"
server_key: "e04c2435d74650a231ec21d70572987c"
server_key_iv: "d28e5677bfd7e2ab160c9a161c54a497"
server_key_version: "encryptionkey@0"
I20260929 01:08:35.268077 15309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:08:35.271474 15323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:35.272830 15309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:08:35.273028 15309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "0a65e545f7cb4a1a92b0323f80f07933"
format_stamp: "Formatted at 2026-09-29 01:08:35 on dist-test-slave-41cl"
server_key: "e04c2435d74650a231ec21d70572987c"
server_key_iv: "d28e5677bfd7e2ab160c9a161c54a497"
server_key_version: "encryptionkey@0"
I20260929 01:08:35.273229 15309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:35.294430 15309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:35.295322 15309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:35.295644 15309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:35.296747 15309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:35.298245 15309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:35.298343 15309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:35.298502 15309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:35.298596 15309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:35.333949 15309 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:37397
I20260929 01:08:35.334046 15435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:37397 every 8 connection(s)
I20260929 01:08:35.335415 15309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:08:35.336937 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 15309
I20260929 01:08:35.337133 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:08:35.337960 5915 external_mini_cluster.cc:1428] Setting key ca660e1ffd6c7a881bc60bfd2f58b256
I20260929 01:08:35.341276 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44003
--builtin_ntp_servers=127.5.198.212:45625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 01:08:35.351281 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:35.338439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:55425 (local address 127.5.198.254:44003)
0929 01:08:35.338881 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:35.338887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:35.340306 (+ 1419us) server_negotiation.cc:407] Connection header received
0929 01:08:35.343056 (+ 2750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:35.343083 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:35.343186 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:35.343278 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:35.344689 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:35.345728 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:35.348190 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:35.348816 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:35.348973 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:35.350451 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:35.350458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:35.350468 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:08:35.350488 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:35.350523 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:08:35.350534 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:35.350624 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:35.350845 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 01:08:35.350950 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":233,"threads_started":1}
I20260929 01:08:35.352326 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:35.339080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44003 (local address 127.5.198.193:55425)
0929 01:08:35.340132 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:35.340189 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:35.342739 (+ 2550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:35.343543 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:35.343557 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:35.343675 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:35.344509 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:35.344536 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:35.345882 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:35.345890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:35.348036 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:35.348059 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:35.349098 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:35.349107 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:35.349343 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:35.350129 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:08:35.350132 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:35.350172 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:08:35.350251 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:35.350636 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:35.350648 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:35.350660 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 01:08:35.350837 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 01:08:35.351250 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":319,"threads_started":1}
I20260929 01:08:35.354048 15436 heartbeater.cc:347] Connected to a master server at 127.5.198.254:44003
I20260929 01:08:35.354344 15436 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:35.355105 15436 heartbeater.cc:511] Master 127.5.198.254:44003 requested a full tablet report, sending...
I20260929 01:08:35.357227 15249 ts_manager.cc:194] Registered new tserver with Master: 0a65e545f7cb4a1a92b0323f80f07933 (127.5.198.193:37397)
I20260929 01:08:35.358839 15249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.198.193:55425
W20260929 01:08:35.661823 15440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:35.662070 15440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:35.662148 15440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:35.669042 15440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:35.669122 15440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:35.669219 15440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:08:35.677805 15440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:45625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:35.678946 15440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:35.680644 15440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:35.693334 15445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:35.693270 15448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:35.693403 15446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:35.693994 15440 server_base.cc:1034] running on GCE node
I20260929 01:08:35.694607 15440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:35.695449 15440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:35.696658 15440 hybrid_clock.cc:648] HybridClock initialized: now 1790644115696615 us; error 50 us; skew 500 ppm
I20260929 01:08:35.696954 15440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:08:35.699126 15440 webserver.cc:460] Webserver started at http://127.5.198.194:35597/ using document root <none> and password file <none>
I20260929 01:08:35.699779 15440 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:35.699905 15440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:35.700209 15440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:35.702926 15440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "6aa618c5ee484c929cd8d5bea2309f83"
format_stamp: "Formatted at 2026-09-29 01:08:35 on dist-test-slave-41cl"
server_key: "aa78e9bbf06eb92ecbff5f3acc3bded1"
server_key_iv: "4bcada4b19773cab7929d1f2186ded69"
server_key_version: "encryptionkey@0"
I20260929 01:08:35.703665 15440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "6aa618c5ee484c929cd8d5bea2309f83"
format_stamp: "Formatted at 2026-09-29 01:08:35 on dist-test-slave-41cl"
server_key: "aa78e9bbf06eb92ecbff5f3acc3bded1"
server_key_iv: "4bcada4b19773cab7929d1f2186ded69"
server_key_version: "encryptionkey@0"
I20260929 01:08:35.708331 15440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 01:08:35.711892 15454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:35.713173 15440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:08:35.713332 15440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "6aa618c5ee484c929cd8d5bea2309f83"
format_stamp: "Formatted at 2026-09-29 01:08:35 on dist-test-slave-41cl"
server_key: "aa78e9bbf06eb92ecbff5f3acc3bded1"
server_key_iv: "4bcada4b19773cab7929d1f2186ded69"
server_key_version: "encryptionkey@0"
I20260929 01:08:35.713555 15440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:35.729943 15440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:35.730674 15440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:35.730966 15440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:35.731876 15440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:35.733201 15440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:35.733317 15440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:35.733450 15440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:35.733534 15440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:35.766667 15440 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:33255
I20260929 01:08:35.766718 15566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:33255 every 8 connection(s)
I20260929 01:08:35.767897 15440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:08:35.775357 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 15440
I20260929 01:08:35.775527 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:08:35.776111 5915 external_mini_cluster.cc:1428] Setting key 8052c391da449304e1d57510e611f4fb
I20260929 01:08:35.778946 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44003
--builtin_ntp_servers=127.5.198.212:45625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 01:08:35.780511 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:35.770290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:56129 (local address 127.5.198.254:44003)
0929 01:08:35.770503 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:35.770510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:35.771804 (+ 1294us) server_negotiation.cc:407] Connection header received
0929 01:08:35.773561 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:35.773566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:35.773631 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:35.773720 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:35.774703 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:35.775450 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:35.777644 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:35.778060 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:35.778190 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:35.779638 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:35.779649 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:35.779653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:08:35.779670 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:35.779701 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:08:35.779712 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:35.779797 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:35.780016 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 01:08:35.780108 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:08:35.781459 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:35.770833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44003 (local address 127.5.198.194:56129)
0929 01:08:35.771660 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:35.771695 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:35.773306 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:35.773946 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:35.773954 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:35.774034 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:35.774566 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:35.774581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:35.775635 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:35.775641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:35.777510 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:35.777521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:35.778270 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:35.778276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:35.778497 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:35.779350 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:08:35.779352 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:35.779390 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:08:35.779461 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:35.779809 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:35.779821 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:35.779828 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 01:08:35.780006 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 01:08:35.780397 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":240,"threads_started":1}
I20260929 01:08:35.782706 15567 heartbeater.cc:347] Connected to a master server at 127.5.198.254:44003
I20260929 01:08:35.783010 15567 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:35.783900 15567 heartbeater.cc:511] Master 127.5.198.254:44003 requested a full tablet report, sending...
I20260929 01:08:35.785404 15249 ts_manager.cc:194] Registered new tserver with Master: 6aa618c5ee484c929cd8d5bea2309f83 (127.5.198.194:33255)
I20260929 01:08:35.786183 15249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.198.194:56129
W20260929 01:08:36.114696 15570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:36.114998 15570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:36.115147 15570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:36.122665 15570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:36.122792 15570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:36.122960 15570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:08:36.131870 15570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:45625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:36.133136 15570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:36.134783 15570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:36.147519 15578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:36.147560 15576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:36.147639 15575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:36.152226 15570 server_base.cc:1034] running on GCE node
I20260929 01:08:36.152913 15570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:36.153810 15570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:36.155051 15570 hybrid_clock.cc:648] HybridClock initialized: now 1790644116155022 us; error 70 us; skew 500 ppm
I20260929 01:08:36.155396 15570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:08:36.157606 15570 webserver.cc:460] Webserver started at http://127.5.198.195:37303/ using document root <none> and password file <none>
I20260929 01:08:36.158290 15570 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:36.158418 15570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:36.158722 15570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:36.161547 15570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "735410b692714128937d179a493483b5"
format_stamp: "Formatted at 2026-09-29 01:08:36 on dist-test-slave-41cl"
server_key: "6f199bfc5b4b599a7f90f93c7f1e75ef"
server_key_iv: "79258807085cdf780f71a0b0517581bd"
server_key_version: "encryptionkey@0"
I20260929 01:08:36.162292 15570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "735410b692714128937d179a493483b5"
format_stamp: "Formatted at 2026-09-29 01:08:36 on dist-test-slave-41cl"
server_key: "6f199bfc5b4b599a7f90f93c7f1e75ef"
server_key_iv: "79258807085cdf780f71a0b0517581bd"
server_key_version: "encryptionkey@0"
I20260929 01:08:36.167097 15570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:08:36.170269 15584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:36.171568 15570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:08:36.171729 15570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "735410b692714128937d179a493483b5"
format_stamp: "Formatted at 2026-09-29 01:08:36 on dist-test-slave-41cl"
server_key: "6f199bfc5b4b599a7f90f93c7f1e75ef"
server_key_iv: "79258807085cdf780f71a0b0517581bd"
server_key_version: "encryptionkey@0"
I20260929 01:08:36.171922 15570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:36.187194 15570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:36.188007 15570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:36.188285 15570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:36.189196 15570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:36.190502 15570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:36.190623 15570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:36.190743 15570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:36.190826 15570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:36.223598 15570 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:44309
I20260929 01:08:36.223680 15696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:44309 every 8 connection(s)
I20260929 01:08:36.224999 15570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:08:36.232326 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 15570
I20260929 01:08:36.232555 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:08:36.232966 5915 external_mini_cluster.cc:1428] Setting key 4533b1d6716173b055bad31655345fc5
I20260929 01:08:36.237910 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:36.227537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:46999 (local address 127.5.198.254:44003)
0929 01:08:36.227704 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:36.227709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:36.229091 (+ 1382us) server_negotiation.cc:407] Connection header received
0929 01:08:36.230957 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:36.230961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:36.231059 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:36.231174 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:36.232520 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:36.233312 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:36.235062 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:36.235679 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:36.235840 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:36.237210 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:36.237215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:36.237219 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:08:36.237235 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:36.237282 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:08:36.237300 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:36.237385 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:36.237595 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 01:08:36.237675 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:08:36.238826 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:36.227998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44003 (local address 127.5.198.195:46999)
0929 01:08:36.228939 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:36.228975 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:36.230584 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:36.231391 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:36.231399 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:36.231483 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:36.232287 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:36.232318 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:36.233489 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:36.233495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:36.234940 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:36.234952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:36.235936 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:36.235944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:36.236135 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:36.236907 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:08:36.236909 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:36.236947 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:08:36.237020 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:36.237393 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:36.237406 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:36.237412 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 01:08:36.237551 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 01:08:36.237817 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":262,"threads_started":1}
I20260929 01:08:36.240072 15697 heartbeater.cc:347] Connected to a master server at 127.5.198.254:44003
I20260929 01:08:36.240362 15697 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:36.241091 15697 heartbeater.cc:511] Master 127.5.198.254:44003 requested a full tablet report, sending...
I20260929 01:08:36.242404 15249 ts_manager.cc:194] Registered new tserver with Master: 735410b692714128937d179a493483b5 (127.5.198.195:44309)
I20260929 01:08:36.243160 15249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.198.195:46999
I20260929 01:08:36.247705 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 01:08:36.253535 15438 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:36.252041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.5.198.254:44003)
0929 01:08:36.252190 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:36.252195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:36.252691 (+ 496us) server_negotiation.cc:407] Connection header received
0929 01:08:36.252811 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:36.252815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:36.252880 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:36.252958 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:36.253293 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60292: BlockingRecv error: recv got EOF from 127.0.0.1:60292 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:08:36.254552 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 15309
I20260929 01:08:36.261904 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:36.255335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.5.198.193:37397)
0929 01:08:36.255831 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:36.255844 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:36.255868 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:08:36.255959 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:36.255965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:36.256158 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:36.256291 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:36.256825 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:36.257835 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:36.259052 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:36.259598 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:36.259690 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:36.260675 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:36.260683 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:36.260694 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:08:36.261228 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:36.261270 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:08:36.261288 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:36.261374 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:36.261519 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 01:08:36.261632 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":278,"threads_started":1}
I20260929 01:08:36.263001 15431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:36.343199 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 15309
I20260929 01:08:36.359432 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 15440
I20260929 01:08:36.366969 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:36.360296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.5.198.194:33255)
0929 01:08:36.360847 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:36.360855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:36.360881 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:08:36.360941 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:36.360947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:36.361162 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:36.361286 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:36.361655 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:36.362637 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:36.363824 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:36.364377 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:36.364655 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:36.365625 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:36.365633 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:36.365645 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:08:36.366201 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:36.366243 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:08:36.366272 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:36.366360 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:36.366501 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 01:08:36.366620 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":300,"threads_started":1}
I20260929 01:08:36.368204 15562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:36.448041 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 15440
I20260929 01:08:36.464412 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 15570
I20260929 01:08:36.472360 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:36.465307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.5.198.195:44309)
0929 01:08:36.465871 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:36.465885 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:36.465914 (+ 29us) server_negotiation.cc:407] Connection header received
0929 01:08:36.465971 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:36.465977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:36.466189 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:36.466365 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:36.466786 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:36.467794 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:36.469101 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:36.469641 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:36.469793 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:36.470990 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:36.470998 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:36.471010 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:08:36.471571 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:36.471616 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:08:36.471633 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:36.471760 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:36.471900 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 01:08:36.472022 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":274,"threads_started":1}
I20260929 01:08:36.473703 15692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:36.555968 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 15570
I20260929 01:08:36.571826 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 15218
I20260929 01:08:36.573055 15279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:36.673308 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 15218
2026-09-29T01:08:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2368 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T01:08:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:08:36Z Disabled control of system clock
I20260929 01:08:36.743602 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:37015
--webserver_interface=127.5.198.254
--webserver_port=0
--builtin_ntp_servers=127.5.198.212:36345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:37015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 01:08:37.062711 15716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:37.062994 15716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:37.063074 15716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:37.069975 15716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:08:37.070071 15716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:37.070108 15716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:37.070154 15716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:08:37.070197 15716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:08:37.079241 15716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:36345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:37015
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:37015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:37.080420 15716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:37.082320 15716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:37.093613 15721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:37.093703 15722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:37.093818 15724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:37.094435 15716 server_base.cc:1034] running on GCE node
I20260929 01:08:37.095139 15716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:37.096428 15716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:37.097680 15716 hybrid_clock.cc:648] HybridClock initialized: now 1790644117097629 us; error 40 us; skew 500 ppm
I20260929 01:08:37.098054 15716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:08:37.100220 15716 webserver.cc:460] Webserver started at http://127.5.198.254:37825/ using document root <none> and password file <none>
I20260929 01:08:37.100968 15716 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:37.101058 15716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:37.101373 15716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:37.104103 15716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "35f3a676c7b64c6f86da201001f65098"
format_stamp: "Formatted at 2026-09-29 01:08:37 on dist-test-slave-41cl"
server_key: "7dddb5839c6c5c8a1cd7158017faad81"
server_key_iv: "86069f436b5efe870cb19e9662ff8898"
server_key_version: "encryptionkey@0"
I20260929 01:08:37.104905 15716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "35f3a676c7b64c6f86da201001f65098"
format_stamp: "Formatted at 2026-09-29 01:08:37 on dist-test-slave-41cl"
server_key: "7dddb5839c6c5c8a1cd7158017faad81"
server_key_iv: "86069f436b5efe870cb19e9662ff8898"
server_key_version: "encryptionkey@0"
I20260929 01:08:37.109575 15716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:08:37.112843 15730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:37.114135 15716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:08:37.114282 15716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "35f3a676c7b64c6f86da201001f65098"
format_stamp: "Formatted at 2026-09-29 01:08:37 on dist-test-slave-41cl"
server_key: "7dddb5839c6c5c8a1cd7158017faad81"
server_key_iv: "86069f436b5efe870cb19e9662ff8898"
server_key_version: "encryptionkey@0"
I20260929 01:08:37.114477 15716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:37.120332 15716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:37.121080 15716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:37.121362 15716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:37.139082 15716 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:37015
I20260929 01:08:37.139122 15781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:37015 every 8 connection(s)
I20260929 01:08:37.140329 15716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:08:37.142567 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 15716
I20260929 01:08:37.142776 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:08:37.143405 5915 external_mini_cluster.cc:1428] Setting key 57f79fa9b64676a036fd3faa3dd087ab
I20260929 01:08:37.144052 15782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:37.152456 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:37.145336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.5.198.254:37015)
0929 01:08:37.146398 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:37.146413 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:37.146460 (+ 47us) server_negotiation.cc:407] Connection header received
0929 01:08:37.148654 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:37.148680 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:37.149192 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:37.149635 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:37.150592 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:37.150605 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:37.150620 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:08:37.150642 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:37.150711 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:08:37.150731 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:37.150943 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:37.151061 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 01:08:37.151322 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":321,"threads_started":1}
I20260929 01:08:37.157162 15782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098: Bootstrap starting.
I20260929 01:08:37.159758 15782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:37.160713 15782 log.cc:824] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098: Log is configured to *not* fsync() on all Append() calls
I20260929 01:08:37.163161 15782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098: No bootstrap required, opened a new log
I20260929 01:08:37.169996 15782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f3a676c7b64c6f86da201001f65098" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37015 } }
I20260929 01:08:37.170315 15782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:37.170406 15782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f3a676c7b64c6f86da201001f65098, State: Initialized, Role: FOLLOWER
I20260929 01:08:37.170935 15782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f3a676c7b64c6f86da201001f65098" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37015 } }
I20260929 01:08:37.171156 15782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:08:37.171284 15782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:08:37.171445 15782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:37.173512 15782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f3a676c7b64c6f86da201001f65098" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37015 } }
I20260929 01:08:37.173970 15782 leader_election.cc:302] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35f3a676c7b64c6f86da201001f65098; no voters:
I20260929 01:08:37.174515 15782 leader_election.cc:288] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:08:37.174659 15787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:08:37.175310 15787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [term 1 LEADER]: Becoming Leader. State: Replica: 35f3a676c7b64c6f86da201001f65098, State: Running, Role: LEADER
I20260929 01:08:37.175769 15787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f3a676c7b64c6f86da201001f65098" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37015 } }
I20260929 01:08:37.176422 15782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:08:37.178493 15789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35f3a676c7b64c6f86da201001f65098. Latest consensus state: current_term: 1 leader_uuid: "35f3a676c7b64c6f86da201001f65098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f3a676c7b64c6f86da201001f65098" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37015 } } }
I20260929 01:08:37.179567 15789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:37.184024 15788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35f3a676c7b64c6f86da201001f65098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f3a676c7b64c6f86da201001f65098" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37015 } } }
I20260929 01:08:37.184437 15788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098 [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:37.185496 15795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:08:37.190186 15795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:08:37.197911 15795 catalog_manager.cc:1347] Generated new cluster ID: 685febaa4cac4c6d80d455522e4403a2
I20260929 01:08:37.198016 15795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:08:37.239711 15795 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:08:37.240145 15795 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:08:37.247956 15795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35f3a676c7b64c6f86da201001f65098: Generated new TSK 0
I20260929 01:08:37.248762 15795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:08:37.257136 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37015
--builtin_ntp_servers=127.5.198.212:36345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 01:08:37.576130 15806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:37.576442 15806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:37.576669 15806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:37.583698 15806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:37.583827 15806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:37.584012 15806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:08:37.593367 15806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:36345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:37.594722 15806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:37.596956 15806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:37.609546 15811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:37.609638 15814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:37.609599 15812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:37.610970 15806 server_base.cc:1034] running on GCE node
I20260929 01:08:37.611531 15806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:37.612483 15806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:37.613688 15806 hybrid_clock.cc:648] HybridClock initialized: now 1790644117613650 us; error 37 us; skew 500 ppm
I20260929 01:08:37.613948 15806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:08:37.616039 15806 webserver.cc:460] Webserver started at http://127.5.198.193:46283/ using document root <none> and password file <none>
I20260929 01:08:37.616703 15806 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:37.616786 15806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:37.617051 15806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:37.619921 15806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "854f7bc7daa041a98f73de9f7e0d69d4"
format_stamp: "Formatted at 2026-09-29 01:08:37 on dist-test-slave-41cl"
server_key: "0b0b7ed6d06f32415e26a31294c5109a"
server_key_iv: "0a7e9d710ccd3b9e0928ee622f82fdb9"
server_key_version: "encryptionkey@0"
I20260929 01:08:37.620687 15806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "854f7bc7daa041a98f73de9f7e0d69d4"
format_stamp: "Formatted at 2026-09-29 01:08:37 on dist-test-slave-41cl"
server_key: "0b0b7ed6d06f32415e26a31294c5109a"
server_key_iv: "0a7e9d710ccd3b9e0928ee622f82fdb9"
server_key_version: "encryptionkey@0"
I20260929 01:08:37.625764 15806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 01:08:37.628973 15820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:37.630514 15806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:08:37.630743 15806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "854f7bc7daa041a98f73de9f7e0d69d4"
format_stamp: "Formatted at 2026-09-29 01:08:37 on dist-test-slave-41cl"
server_key: "0b0b7ed6d06f32415e26a31294c5109a"
server_key_iv: "0a7e9d710ccd3b9e0928ee622f82fdb9"
server_key_version: "encryptionkey@0"
I20260929 01:08:37.630949 15806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:37.636629 15806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:37.637477 15806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:37.637769 15806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:37.638700 15806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:37.640137 15806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:37.640268 15806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:37.640401 15806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:37.640569 15806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:37.674554 15806 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:41009
I20260929 01:08:37.674634 15932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:41009 every 8 connection(s)
I20260929 01:08:37.675973 15806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:08:37.679971 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 15806
I20260929 01:08:37.680186 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:08:37.681054 5915 external_mini_cluster.cc:1428] Setting key 212154fcfa45186b740c8938beef3ab0
I20260929 01:08:37.684228 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37015
--builtin_ntp_servers=127.5.198.212:36345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 01:08:37.685359 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:37.678674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:41451 (local address 127.5.198.254:37015)
0929 01:08:37.679142 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:37.679147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:37.680659 (+ 1512us) server_negotiation.cc:407] Connection header received
0929 01:08:37.682766 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:37.682775 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:37.682835 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:37.682991 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:37.684541 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:37.684550 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:37.684557 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:08:37.684580 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:37.684630 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:08:37.684647 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:37.684736 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:37.685078 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 01:08:37.685142 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":244,"threads_started":1}
I20260929 01:08:37.686558 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:37.679308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37015 (local address 127.5.198.193:41451)
0929 01:08:37.680458 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:37.680537 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:37.682499 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:37.683229 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:37.683240 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:37.683388 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:37.684129 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:08:37.684133 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:37.684181 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:08:37.684306 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:37.684828 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:37.684840 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:37.684849 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 01:08:37.685116 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 01:08:37.685375 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":313,"threads_started":1}
I20260929 01:08:37.688001 15933 heartbeater.cc:347] Connected to a master server at 127.5.198.254:37015
I20260929 01:08:37.688340 15933 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:37.689009 15933 heartbeater.cc:511] Master 127.5.198.254:37015 requested a full tablet report, sending...
I20260929 01:08:37.691222 15747 ts_manager.cc:194] Registered new tserver with Master: 854f7bc7daa041a98f73de9f7e0d69d4 (127.5.198.193:41009)
W20260929 01:08:38.005389 15937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:38.005640 15937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:38.005807 15937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:38.012892 15937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:38.012993 15937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:38.013162 15937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:08:38.022274 15937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:36345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:38.023535 15937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:38.025445 15937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:38.038652 15943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:38.038657 15942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:38.038691 15945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:38.039569 15937 server_base.cc:1034] running on GCE node
I20260929 01:08:38.040226 15937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:38.041261 15937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:38.042467 15937 hybrid_clock.cc:648] HybridClock initialized: now 1790644118042399 us; error 61 us; skew 500 ppm
I20260929 01:08:38.042778 15937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:08:38.045480 15937 webserver.cc:460] Webserver started at http://127.5.198.194:34747/ using document root <none> and password file <none>
I20260929 01:08:38.046197 15937 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:38.046314 15937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:38.046675 15937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:38.049760 15937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "c59cbc212d3941aa86e0fc5137636a9b"
format_stamp: "Formatted at 2026-09-29 01:08:38 on dist-test-slave-41cl"
server_key: "207883057ba2e6868e7a5036844c84fd"
server_key_iv: "8227a36b3089428846298c9a5abc31f5"
server_key_version: "encryptionkey@0"
I20260929 01:08:38.050608 15937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "c59cbc212d3941aa86e0fc5137636a9b"
format_stamp: "Formatted at 2026-09-29 01:08:38 on dist-test-slave-41cl"
server_key: "207883057ba2e6868e7a5036844c84fd"
server_key_iv: "8227a36b3089428846298c9a5abc31f5"
server_key_version: "encryptionkey@0"
I20260929 01:08:38.055845 15937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:08:38.059437 15951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:38.060932 15937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:08:38.061100 15937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "c59cbc212d3941aa86e0fc5137636a9b"
format_stamp: "Formatted at 2026-09-29 01:08:38 on dist-test-slave-41cl"
server_key: "207883057ba2e6868e7a5036844c84fd"
server_key_iv: "8227a36b3089428846298c9a5abc31f5"
server_key_version: "encryptionkey@0"
I20260929 01:08:38.061311 15937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:38.066797 15937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:38.067567 15937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:38.067821 15937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:38.068876 15937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:38.070182 15937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:38.070300 15937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:38.070430 15937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:38.070516 15937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:38.103037 15937 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:33455
I20260929 01:08:38.103089 16063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:33455 every 8 connection(s)
I20260929 01:08:38.104326 15937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:08:38.107028 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 15937
I20260929 01:08:38.107208 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:08:38.107632 5915 external_mini_cluster.cc:1428] Setting key 0a52a92f5188ccaca4507a1cae66aed7
I20260929 01:08:38.110227 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37015
--builtin_ntp_servers=127.5.198.212:36345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 01:08:38.112699 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:38.106931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:39519 (local address 127.5.198.254:37015)
0929 01:08:38.107151 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:38.107159 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:38.108768 (+ 1609us) server_negotiation.cc:407] Connection header received
0929 01:08:38.110425 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:38.110434 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:38.110507 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:38.110606 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:38.111931 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:38.111938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:38.111946 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:08:38.111968 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:38.112007 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:08:38.112025 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:38.112144 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:38.112285 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 01:08:38.112345 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 01:08:38.113484 16066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:38.107508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37015 (local address 127.5.198.194:39519)
0929 01:08:38.108609 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:38.108661 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:38.110135 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:38.110815 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:38.110823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:38.110957 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:38.111641 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:08:38.111644 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:38.111680 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:08:38.111762 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:38.112113 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:38.112118 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:38.112123 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 01:08:38.112315 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 01:08:38.112559 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":361,"threads_started":1}
I20260929 01:08:38.114789 16064 heartbeater.cc:347] Connected to a master server at 127.5.198.254:37015
I20260929 01:08:38.115080 16064 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:38.115650 16064 heartbeater.cc:511] Master 127.5.198.254:37015 requested a full tablet report, sending...
I20260929 01:08:38.117064 15747 ts_manager.cc:194] Registered new tserver with Master: c59cbc212d3941aa86e0fc5137636a9b (127.5.198.194:33455)
W20260929 01:08:38.439926 16067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:38.440235 16067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:38.440325 16067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:38.447333 16067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:38.447423 16067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:38.447530 16067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:08:38.456264 16067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:36345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:38.457526 16067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:38.459196 16067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:38.471851 16072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:38.471835 16073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:38.471855 16075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:38.472587 16067 server_base.cc:1034] running on GCE node
I20260929 01:08:38.473193 16067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:38.474026 16067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:38.475208 16067 hybrid_clock.cc:648] HybridClock initialized: now 1790644118475154 us; error 51 us; skew 500 ppm
I20260929 01:08:38.475500 16067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:08:38.477862 16067 webserver.cc:460] Webserver started at http://127.5.198.195:43879/ using document root <none> and password file <none>
I20260929 01:08:38.478518 16067 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:38.478644 16067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:38.478961 16067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:38.481681 16067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "741a9ed3a32b4196b8bf15bb0c06be34"
format_stamp: "Formatted at 2026-09-29 01:08:38 on dist-test-slave-41cl"
server_key: "c1a479b011ab7e64898b5a1f319b06d1"
server_key_iv: "262f2b131ecef258f1869d0f1519c783"
server_key_version: "encryptionkey@0"
I20260929 01:08:38.482431 16067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "741a9ed3a32b4196b8bf15bb0c06be34"
format_stamp: "Formatted at 2026-09-29 01:08:38 on dist-test-slave-41cl"
server_key: "c1a479b011ab7e64898b5a1f319b06d1"
server_key_iv: "262f2b131ecef258f1869d0f1519c783"
server_key_version: "encryptionkey@0"
I20260929 01:08:38.487262 16067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 01:08:38.490797 16081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:38.492660 16067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:08:38.492851 16067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "741a9ed3a32b4196b8bf15bb0c06be34"
format_stamp: "Formatted at 2026-09-29 01:08:38 on dist-test-slave-41cl"
server_key: "c1a479b011ab7e64898b5a1f319b06d1"
server_key_iv: "262f2b131ecef258f1869d0f1519c783"
server_key_version: "encryptionkey@0"
I20260929 01:08:38.493120 16067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:38.498729 16067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:38.499467 16067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:38.499715 16067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:38.500618 16067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:38.501953 16067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:38.502056 16067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:38.502170 16067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:38.502251 16067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:38.533061 16067 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:37363
I20260929 01:08:38.533138 16193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:37363 every 8 connection(s)
I20260929 01:08:38.534403 16067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:08:38.543021 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:38.537515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:44475 (local address 127.5.198.254:37015)
0929 01:08:38.537690 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:38.537694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:38.539090 (+ 1396us) server_negotiation.cc:407] Connection header received
0929 01:08:38.540817 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:38.540823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:38.540891 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:38.541003 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:38.542264 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:38.542270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:38.542274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:08:38.542293 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:38.542330 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:08:38.542341 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:38.542488 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:38.542657 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 01:08:38.542722 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:08:38.543421 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 16067
I20260929 01:08:38.543601 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:08:38.544000 5915 external_mini_cluster.cc:1428] Setting key eb8e539a3b81544ea3a170351bb12cfb
I20260929 01:08:38.543989 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:38.538033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37015 (local address 127.5.198.195:44475)
0929 01:08:38.538926 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:38.538968 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:38.540558 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:38.541215 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:38.541223 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:38.541312 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:38.541977 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:08:38.541979 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:38.542015 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:08:38.542095 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:38.542448 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:38.542458 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:38.542466 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 01:08:38.542738 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 01:08:38.542990 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":282,"threads_started":1}
I20260929 01:08:38.545681 16194 heartbeater.cc:347] Connected to a master server at 127.5.198.254:37015
I20260929 01:08:38.546034 16194 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:38.546598 16194 heartbeater.cc:511] Master 127.5.198.254:37015 requested a full tablet report, sending...
I20260929 01:08:38.547930 15747 ts_manager.cc:194] Registered new tserver with Master: 741a9ed3a32b4196b8bf15bb0c06be34 (127.5.198.195:37363)
I20260929 01:08:38.559113 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 01:08:38.565122 15935 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:08:38.563603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.5.198.254:37015)
0929 01:08:38.563757 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:38.563761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:38.564228 (+ 467us) server_negotiation.cc:407] Connection header received
0929 01:08:38.564308 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:38.564311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:38.564359 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:38.564448 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:38.564864 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40362: BlockingRecv error: recv got EOF from 127.0.0.1:40362 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:08:38.565543 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 15806
I20260929 01:08:38.568910 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:38.566471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.5.198.193:41009)
0929 01:08:38.566948 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:38.566965 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:38.566998 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:08:38.567051 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:38.567057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:38.567248 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:38.567376 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:38.568232 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:38.568238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:38.568249 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:08:38.568266 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:38.568302 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:08:38.568316 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:38.568426 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:38.568613 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 01:08:38.568692 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":237,"threads_started":1}
I20260929 01:08:38.570035 15928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:38.651321 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 15806
I20260929 01:08:38.669509 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 15937
I20260929 01:08:38.673041 16205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:38.670447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.5.198.194:33455)
0929 01:08:38.670911 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:38.670917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:38.670950 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:08:38.671009 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:38.671015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:38.671228 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:38.671391 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:38.672344 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:38.672351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:38.672362 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:08:38.672381 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:38.672420 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:08:38.672434 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:38.672600 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:38.672681 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 01:08:38.672777 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":219,"threads_started":1}
I20260929 01:08:38.674244 16059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:38.756013 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 15937
I20260929 01:08:38.772199 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 16067
I20260929 01:08:38.776337 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:38.773215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42418 (local address 127.5.198.195:37363)
0929 01:08:38.773886 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:38.773893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:38.773926 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:08:38.773989 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:38.773996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:38.774214 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:38.774404 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:38.775543 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:38.775549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:38.775560 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:08:38.775576 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:38.775614 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:08:38.775627 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:38.775806 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:38.775891 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 01:08:38.776013 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":322,"threads_started":1}
I20260929 01:08:38.777627 16189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:38.859329 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 16067
I20260929 01:08:38.877058 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 15716
I20260929 01:08:38.878363 15777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:38.979630 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 15716
2026-09-29T01:08:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2305 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:08:39 dist-test-slave-41cl krb5kdc[16212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:08:39 dist-test-slave-41cl krb5kdc[16212](info): set up 2 sockets
Sep 29 01:08:39 dist-test-slave-41cl krb5kdc[16212](info): commencing operation
krb5kdc: starting...
W20260929 01:08:40.085420 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:08:40 dist-test-slave-41cl krb5kdc[16212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:08:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:08:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:08:40.324892 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:33123
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:33123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:08:40.641458 16225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:40.641773 16225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:40.641918 16225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:40.649060 16225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:08:40.649186 16225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:40.649290 16225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:40.649395 16225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:08:40.649468 16225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:08:40.659140 16225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:33123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:33123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:40.660444 16225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:40.661794 16225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:40.673251 16230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:40.673251 16231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:40.673475 16233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:40.674551 16225 server_base.cc:1034] running on GCE node
I20260929 01:08:40.675340 16225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:40.676808 16225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:40.678028 16225 hybrid_clock.cc:648] HybridClock initialized: now 1790644120677967 us; error 51 us; skew 500 ppm
I20260929 01:08:40.678395 16225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:40 dist-test-slave-41cl krb5kdc[16212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644120, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:40.688586 16225 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:08:40.690198 16225 webserver.cc:460] Webserver started at http://127.5.198.254:39243/ using document root <none> and password file <none>
I20260929 01:08:40.690907 16225 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:40.691017 16225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:40.691286 16225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:40.694478 16225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "4f869c8e746d4801ae2a9a439156efcc"
format_stamp: "Formatted at 2026-09-29 01:08:40 on dist-test-slave-41cl"
server_key: "3cc401218df64a4528609064cd2576b0"
server_key_iv: "a60133f61429a067011a0be700d301cd"
server_key_version: "encryptionkey@0"
I20260929 01:08:40.695236 16225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "4f869c8e746d4801ae2a9a439156efcc"
format_stamp: "Formatted at 2026-09-29 01:08:40 on dist-test-slave-41cl"
server_key: "3cc401218df64a4528609064cd2576b0"
server_key_iv: "a60133f61429a067011a0be700d301cd"
server_key_version: "encryptionkey@0"
I20260929 01:08:40.700632 16225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:08:40.704185 16240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:40.705647 16225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:08:40.705821 16225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "4f869c8e746d4801ae2a9a439156efcc"
format_stamp: "Formatted at 2026-09-29 01:08:40 on dist-test-slave-41cl"
server_key: "3cc401218df64a4528609064cd2576b0"
server_key_iv: "a60133f61429a067011a0be700d301cd"
server_key_version: "encryptionkey@0"
I20260929 01:08:40.706017 16225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:40.725132 16225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:40.729210 16225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:40.729485 16225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:40.747605 16225 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:33123
I20260929 01:08:40.747634 16291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:33123 every 8 connection(s)
I20260929 01:08:40.748996 16225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:08:40.752501 16292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:40.756352 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 16225
I20260929 01:08:40.756609 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:08:40.757364 5915 external_mini_cluster.cc:1428] Setting key 16ee2b0ba7dc606f024aba4ee70f5c9a
I20260929 01:08:40.763619 16292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc: Bootstrap starting.
I20260929 01:08:40.767024 16292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:40.768151 16292 log.cc:824] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc: Log is configured to *not* fsync() on all Append() calls
I20260929 01:08:40.771502 16292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc: No bootstrap required, opened a new log
Sep 29 01:08:40 dist-test-slave-41cl krb5kdc[16212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:08:40.781422 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:40.759344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42102 (local address 127.5.198.254:33123)
0929 01:08:40.760091 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:40.760101 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:40.760139 (+ 38us) server_negotiation.cc:407] Connection header received
0929 01:08:40.762219 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:40.762241 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:40.762604 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:40.762913 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:40.764631 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:40.766049 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:40.767817 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:40.768590 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:40.768814 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:40.773209 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:40.773220 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:40.773244 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:40.773286 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:40.777271 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:40.777918 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:40.777931 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:40.777946 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:40.778137 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:40.778464 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:40.778470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:40.778479 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:40.779201 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:40.779477 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:40.779780 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 01:08:40.780230 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":340,"threads_started":1}
I20260929 01:08:40.781791 16292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f869c8e746d4801ae2a9a439156efcc" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33123 } }
I20260929 01:08:40.782153 16292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:40.782263 16292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f869c8e746d4801ae2a9a439156efcc, State: Initialized, Role: FOLLOWER
I20260929 01:08:40.782788 16292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f869c8e746d4801ae2a9a439156efcc" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33123 } }
I20260929 01:08:40.783030 16292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:08:40.783161 16292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:08:40.783340 16292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:40.785655 16292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f869c8e746d4801ae2a9a439156efcc" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33123 } }
I20260929 01:08:40.786106 16292 leader_election.cc:302] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f869c8e746d4801ae2a9a439156efcc; no voters:
I20260929 01:08:40.786666 16292 leader_election.cc:288] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:08:40.786794 16297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:08:40.787448 16297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [term 1 LEADER]: Becoming Leader. State: Replica: 4f869c8e746d4801ae2a9a439156efcc, State: Running, Role: LEADER
I20260929 01:08:40.788033 16297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f869c8e746d4801ae2a9a439156efcc" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33123 } }
I20260929 01:08:40.788568 16292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:08:40.790576 16298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f869c8e746d4801ae2a9a439156efcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f869c8e746d4801ae2a9a439156efcc" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33123 } } }
I20260929 01:08:40.790921 16298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:40.791440 16299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f869c8e746d4801ae2a9a439156efcc. Latest consensus state: current_term: 1 leader_uuid: "4f869c8e746d4801ae2a9a439156efcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f869c8e746d4801ae2a9a439156efcc" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33123 } } }
I20260929 01:08:40.791745 16299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc [sys.catalog]: This master's current role is: LEADER
I20260929 01:08:40.792675 16303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:08:40.797768 16303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:08:40.806849 16303 catalog_manager.cc:1347] Generated new cluster ID: 777ba018ab124c8e97c375a3735e4ed7
I20260929 01:08:40.807044 16303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:08:40.818670 16303 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:08:40.820046 16303 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:08:40.831609 16303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc: Generated new TSK 0
I20260929 01:08:40.832682 16303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:08:40.892844 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33123
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:08:41.216992 16318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:41.217258 16318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:41.217347 16318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:41.224380 16318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:41.224493 16318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:41.224656 16318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:08:41.234372 16318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:41.235611 16318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:41.236898 16318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:41.250037 16323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:41.250002 16326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:41.249972 16324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:41.250847 16318 server_base.cc:1034] running on GCE node
I20260929 01:08:41.251402 16318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:41.252254 16318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:41.253434 16318 hybrid_clock.cc:648] HybridClock initialized: now 1790644121253379 us; error 52 us; skew 500 ppm
I20260929 01:08:41.253765 16318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:41 dist-test-slave-41cl krb5kdc[16212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:41.256955 16318 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:08:41.258322 16318 webserver.cc:460] Webserver started at http://127.5.198.193:39641/ using document root <none> and password file <none>
I20260929 01:08:41.259045 16318 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:41.259183 16318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:41.259472 16318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:41.262260 16318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "23ad6e9449a348e282e0fc342991d50b"
format_stamp: "Formatted at 2026-09-29 01:08:41 on dist-test-slave-41cl"
server_key: "9b56a807cf3ef56643479ad920f18f90"
server_key_iv: "318629087923e0a8d2d05fcd7c273751"
server_key_version: "encryptionkey@0"
I20260929 01:08:41.263013 16318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "23ad6e9449a348e282e0fc342991d50b"
format_stamp: "Formatted at 2026-09-29 01:08:41 on dist-test-slave-41cl"
server_key: "9b56a807cf3ef56643479ad920f18f90"
server_key_iv: "318629087923e0a8d2d05fcd7c273751"
server_key_version: "encryptionkey@0"
I20260929 01:08:41.267951 16318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:08:41.271283 16333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:41.272677 16318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:08:41.272837 16318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "23ad6e9449a348e282e0fc342991d50b"
format_stamp: "Formatted at 2026-09-29 01:08:41 on dist-test-slave-41cl"
server_key: "9b56a807cf3ef56643479ad920f18f90"
server_key_iv: "318629087923e0a8d2d05fcd7c273751"
server_key_version: "encryptionkey@0"
I20260929 01:08:41.273025 16318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:41.284025 16318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:41.287921 16318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:41.288216 16318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:41.289218 16318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:41.290714 16318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:41.290935 16318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:41.291090 16318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:41.291160 16318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:41.324165 16318 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:45225
I20260929 01:08:41.324200 16445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:45225 every 8 connection(s)
I20260929 01:08:41.325482 16318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:08:41.334925 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 16318
I20260929 01:08:41.335150 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:08:41.335927 5915 external_mini_cluster.cc:1428] Setting key b17c822de514df4c696db0f30adba5ba
Sep 29 01:08:41 dist-test-slave-41cl krb5kdc[16212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:08:41.346987 16448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:41.327966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:49545 (local address 127.5.198.254:33123)
0929 01:08:41.328379 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:41.328383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:41.329411 (+ 1028us) server_negotiation.cc:407] Connection header received
0929 01:08:41.331202 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:41.331209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:41.331305 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:41.331447 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:41.333100 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:41.333799 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:41.335480 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:41.336134 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:41.336272 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:41.340108 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:41.340118 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:41.340126 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:41.340176 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:41.343725 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:41.344539 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:41.344547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:41.344554 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:41.344697 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:41.345138 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:41.345145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:41.345152 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:41.345517 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:41.345670 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:41.346374 (+ 704us) server_negotiation.cc:299] Negotiation successful
0929 01:08:41.346601 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":211,"threads_started":1}
I20260929 01:08:41.347973 16449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:41.328551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33123 (local address 127.5.198.193:49545)
0929 01:08:41.329251 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:41.329290 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:41.330911 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:41.331692 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:41.331701 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:41.332302 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:41.332928 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:41.332946 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:41.333951 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:41.333955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:41.335312 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:41.335324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:41.336336 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:41.336341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:41.336629 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:41.337311 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:41.337340 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:41.339884 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:41.343878 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:41.343885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:41.343902 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:41.344363 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:41.344813 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:41.344817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:41.344819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:41.345022 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:41.345646 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:41.345652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:41.345988 (+ 336us) client_negotiation.cc:769] Sending connection context
0929 01:08:41.346260 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 01:08:41.346811 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":269,"threads_started":1}
I20260929 01:08:41.349666 16446 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33123
I20260929 01:08:41.350005 16446 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:41.350855 16446 heartbeater.cc:511] Master 127.5.198.254:33123 requested a full tablet report, sending...
I20260929 01:08:41.353243 16257 ts_manager.cc:194] Registered new tserver with Master: 23ad6e9449a348e282e0fc342991d50b (127.5.198.193:45225)
I20260929 01:08:41.355345 16257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:49545
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:08:41.391381 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33123
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:08:41.718120 16452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:41.718396 16452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:41.718546 16452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:41.725701 16452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:41.725819 16452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:41.726105 16452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:08:41.736052 16452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:41.737432 16452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:41.738749 16452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:41.751695 16457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:41.752197 16458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:41.752687 16460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:41.752952 16452 server_base.cc:1034] running on GCE node
I20260929 01:08:41.753553 16452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:41.754446 16452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:41.755661 16452 hybrid_clock.cc:648] HybridClock initialized: now 1790644121755610 us; error 45 us; skew 500 ppm
I20260929 01:08:41.755972 16452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:41 dist-test-slave-41cl krb5kdc[16212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:41.759280 16452 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:08:41.760984 16452 webserver.cc:460] Webserver started at http://127.5.198.194:46641/ using document root <none> and password file <none>
I20260929 01:08:41.761830 16452 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:41.761971 16452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:41.762317 16452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:41.766208 16452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "e80b815f61774d989cb32539da227ae3"
format_stamp: "Formatted at 2026-09-29 01:08:41 on dist-test-slave-41cl"
server_key: "01d49e745f6b425828d50c5895c090b7"
server_key_iv: "ec61c417c138a73868e448456194c0bd"
server_key_version: "encryptionkey@0"
I20260929 01:08:41.767143 16452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "e80b815f61774d989cb32539da227ae3"
format_stamp: "Formatted at 2026-09-29 01:08:41 on dist-test-slave-41cl"
server_key: "01d49e745f6b425828d50c5895c090b7"
server_key_iv: "ec61c417c138a73868e448456194c0bd"
server_key_version: "encryptionkey@0"
I20260929 01:08:41.772445 16452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:08:41.775918 16467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:41.777580 16452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:08:41.777742 16452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "e80b815f61774d989cb32539da227ae3"
format_stamp: "Formatted at 2026-09-29 01:08:41 on dist-test-slave-41cl"
server_key: "01d49e745f6b425828d50c5895c090b7"
server_key_iv: "ec61c417c138a73868e448456194c0bd"
server_key_version: "encryptionkey@0"
I20260929 01:08:41.777928 16452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:41.798466 16452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:41.802870 16452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:41.803160 16452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:41.804148 16452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:41.805512 16452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:41.805583 16452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:41.805711 16452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:41.805758 16452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:41.838176 16452 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:46259
I20260929 01:08:41.838256 16579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:46259 every 8 connection(s)
I20260929 01:08:41.839489 16452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:08:41.842689 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 16452
I20260929 01:08:41.842893 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:08:41.843326 5915 external_mini_cluster.cc:1428] Setting key 2bfeb45e7541687202ff2672bfeaba9d
Sep 29 01:08:41 dist-test-slave-41cl krb5kdc[16212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:08:41.862311 16448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:41.842021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:39725 (local address 127.5.198.254:33123)
0929 01:08:41.842178 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:41.842182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:41.843404 (+ 1222us) server_negotiation.cc:407] Connection header received
0929 01:08:41.845816 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:41.845824 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:41.845909 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:41.846009 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:41.847457 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:41.848609 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:41.850239 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:41.850967 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:41.851098 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:41.854597 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:41.854606 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:41.854615 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:41.854660 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:41.858461 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:41.859545 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:41.859553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:41.859560 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:41.859711 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:41.860271 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:41.860276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:41.860280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:41.860653 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:41.860814 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:41.861688 (+ 874us) server_negotiation.cc:299] Negotiation successful
0929 01:08:41.861945 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:08:41.863662 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:41.842453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33123 (local address 127.5.198.194:39725)
0929 01:08:41.843242 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:41.843287 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:41.845554 (+ 2267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:41.846219 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:41.846228 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:41.846741 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:41.847284 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:41.847301 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:41.848742 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:41.848745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:41.850098 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:41.850110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:41.851167 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:41.851172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:41.851352 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:41.852017 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:41.852043 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:41.854323 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:41.858650 (+ 4327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:41.858662 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:41.858681 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:41.859419 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:41.859852 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:41.859859 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:41.859866 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:41.860095 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:41.860799 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:41.860811 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:41.861243 (+ 432us) client_negotiation.cc:769] Sending connection context
0929 01:08:41.861558 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 01:08:41.862122 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":325,"threads_started":1}
I20260929 01:08:41.865237 16580 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33123
I20260929 01:08:41.865571 16580 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:41.866307 16580 heartbeater.cc:511] Master 127.5.198.254:33123 requested a full tablet report, sending...
I20260929 01:08:41.867743 16257 ts_manager.cc:194] Registered new tserver with Master: e80b815f61774d989cb32539da227ae3 (127.5.198.194:46259)
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
I20260929 01:08:41.868623 16257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:39725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:08:41.897921 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33123
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:08:42.220099 16585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:08:42.220368 16585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:08:42.220546 16585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:08:42.227651 16585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:08:42.227753 16585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:08:42.227923 16585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:08:42.237709 16585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:08:42.239063 16585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:08:42.240339 16585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:08:42.252982 16593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:42.253021 16591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:08:42.253043 16590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:08:42.254220 16585 server_base.cc:1034] running on GCE node
I20260929 01:08:42.254770 16585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:08:42.255612 16585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:08:42.256800 16585 hybrid_clock.cc:648] HybridClock initialized: now 1790644122256750 us; error 40 us; skew 500 ppm
I20260929 01:08:42.257087 16585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:08:42 dist-test-slave-41cl krb5kdc[16212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644122, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:08:42.260190 16585 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:08:42.261507 16585 webserver.cc:460] Webserver started at http://127.5.198.195:38163/ using document root <none> and password file <none>
I20260929 01:08:42.262117 16585 fs_manager.cc:359] Metadata directory not provided
I20260929 01:08:42.262212 16585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:08:42.262454 16585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:08:42.265232 16585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "3d5e220f38624321861c416a0ca50a84"
format_stamp: "Formatted at 2026-09-29 01:08:42 on dist-test-slave-41cl"
server_key: "387022e14b2463daf29c375b2c9b7dc8"
server_key_iv: "1b3e0c0c69505f56a4484795a4c68abb"
server_key_version: "encryptionkey@0"
I20260929 01:08:42.265940 16585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "3d5e220f38624321861c416a0ca50a84"
format_stamp: "Formatted at 2026-09-29 01:08:42 on dist-test-slave-41cl"
server_key: "387022e14b2463daf29c375b2c9b7dc8"
server_key_iv: "1b3e0c0c69505f56a4484795a4c68abb"
server_key_version: "encryptionkey@0"
I20260929 01:08:42.270737 16585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:08:42.274070 16600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:42.275388 16585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:08:42.275557 16585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "3d5e220f38624321861c416a0ca50a84"
format_stamp: "Formatted at 2026-09-29 01:08:42 on dist-test-slave-41cl"
server_key: "387022e14b2463daf29c375b2c9b7dc8"
server_key_iv: "1b3e0c0c69505f56a4484795a4c68abb"
server_key_version: "encryptionkey@0"
I20260929 01:08:42.275759 16585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:08:42.293648 16585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:08:42.297946 16585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:08:42.298236 16585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:08:42.299193 16585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:08:42.300529 16585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:08:42.300623 16585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:42.300765 16585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:08:42.300832 16585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:08:42.334192 16585 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:35651
I20260929 01:08:42.334230 16712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:35651 every 8 connection(s)
I20260929 01:08:42.335412 16585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:08:42.339921 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 16585
I20260929 01:08:42.340153 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:08:42.340770 5915 external_mini_cluster.cc:1428] Setting key 125a08cb610e49f0d8b61d7106b157e2
Sep 29 01:08:42 dist-test-slave-41cl krb5kdc[16212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644122, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:08:42.355782 16448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.337936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:50999 (local address 127.5.198.254:33123)
0929 01:08:42.338149 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:42.338155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:42.339426 (+ 1271us) server_negotiation.cc:407] Connection header received
0929 01:08:42.342020 (+ 2594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:42.342029 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:42.342146 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:42.342254 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:42.344148 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.344944 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.346473 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.346848 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.346953 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.350235 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:42.350240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:42.350244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:42.350274 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:42.352690 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:42.353355 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:42.353359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:42.353361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:42.353480 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:42.353910 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:42.353913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:42.353915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:42.354227 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:42.354371 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:42.355177 (+ 806us) server_negotiation.cc:299] Negotiation successful
0929 01:08:42.355405 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 01:08:42.356925 16715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.338459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33123 (local address 127.5.198.195:50999)
0929 01:08:42.339260 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:42.339300 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:42.341690 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:42.342485 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:42.342493 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:42.343273 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:08:42.343985 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:42.344004 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.345110 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.345114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:42.346342 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:42.346354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.347035 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.347046 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:42.347240 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.347861 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:08:42.347881 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:08:42.350054 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:08:42.352845 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:42.352851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:42.352863 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:42.353250 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:42.353599 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:08:42.353603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:08:42.353606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:08:42.353811 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:08:42.354372 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:08:42.354378 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:08:42.354683 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 01:08:42.355083 (+ 400us) client_negotiation.cc:240] Negotiation successful
0929 01:08:42.355602 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":289,"threads_started":1}
I20260929 01:08:42.358212 16713 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33123
I20260929 01:08:42.358505 16713 heartbeater.cc:464] Registering TS with master...
I20260929 01:08:42.359215 16446 heartbeater.cc:503] Master 127.5.198.254:33123 was elected leader, sending a full tablet report...
I20260929 01:08:42.359272 16713 heartbeater.cc:511] Master 127.5.198.254:33123 requested a full tablet report, sending...
I20260929 01:08:42.360767 16256 ts_manager.cc:194] Registered new tserver with Master: 3d5e220f38624321861c416a0ca50a84 (127.5.198.195:35651)
I20260929 01:08:42.361578 16256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:50999
I20260929 01:08:42.368659 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:08:42.385674 16448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.373061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.5.198.254:33123)
0929 01:08:42.373215 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:42.373236 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:42.373698 (+ 462us) server_negotiation.cc:407] Connection header received
0929 01:08:42.373897 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:42.373901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:42.373959 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:42.374054 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:42.375371 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.376222 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.377507 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.377951 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.378077 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.380295 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:42.380300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:42.380303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:42.380334 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:42.383077 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:42.383717 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:42.383727 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:42.383731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:42.383862 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:42.384217 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:42.384226 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:42.384228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:42.384583 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:42.384820 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:42.385148 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 01:08:42.385355 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:08:42.390064 16256 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:08:42.397828 16256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:08:42.425212 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.410364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.5.198.194:46259)
0929 01:08:42.411512 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:42.411519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:42.412576 (+ 1057us) server_negotiation.cc:407] Connection header received
0929 01:08:42.413062 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:42.413071 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:42.413360 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:42.413613 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:42.414821 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.416960 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.421043 (+ 4083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.423083 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.423224 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.423498 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:42.424540 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0929 01:08:42.424777 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":958,"thread_start_us":859,"threads_started":1}
I20260929 01:08:42.425282 16727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.411590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36306 (local address 127.5.198.193:45225)
0929 01:08:42.412275 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:42.412285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:42.412988 (+ 703us) server_negotiation.cc:407] Connection header received
0929 01:08:42.413225 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:42.413233 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:42.413501 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:42.413721 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:42.414985 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.417142 (+ 2157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.421054 (+ 3912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.423069 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.423217 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.423532 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:42.424525 (+ 993us) server_negotiation.cc:299] Negotiation successful
0929 01:08:42.424762 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":485,"threads_started":1}
I20260929 01:08:42.426262 16728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.412064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:45225 (local address 127.0.0.1:36306)
0929 01:08:42.412893 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:42.412910 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:42.413074 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:42.414001 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:42.414008 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:42.414044 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:42.414721 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:42.414737 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.417388 (+ 2651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.417396 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:42.420830 (+ 3434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:42.420850 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.423838 (+ 2988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.423848 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:42.424193 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.424254 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 01:08:42.425513 (+ 1259us) client_negotiation.cc:240] Negotiation successful
0929 01:08:42.425707 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":409,"thread_start_us":348,"threads_started":1}
I20260929 01:08:42.427524 16725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.410274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:46259 (local address 127.0.0.1:55852)
0929 01:08:42.412373 (+ 2099us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:42.412413 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:42.412754 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:42.413925 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:42.413936 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:42.413973 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:42.414566 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:42.414583 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.417202 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.417210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:42.420813 (+ 3603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:42.420835 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.423786 (+ 2951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.423795 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:42.424163 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.424235 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 01:08:42.425839 (+ 1604us) client_negotiation.cc:240] Negotiation successful
0929 01:08:42.426043 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1802,"mutex_wait_us":43,"thread_start_us":1572,"threads_started":1}
I20260929 01:08:42.430282 16381 tablet_service.cc:1467] Processing CreateTablet for tablet e53261a83ffb4e1da6f8fe5a9c043102 (DEFAULT_TABLE table=test-table [id=7d83d80edf924b0f9f5004b216beb6e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:42.431514 16381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53261a83ffb4e1da6f8fe5a9c043102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:42.431207 16515 tablet_service.cc:1467] Processing CreateTablet for tablet e53261a83ffb4e1da6f8fe5a9c043102 (DEFAULT_TABLE table=test-table [id=7d83d80edf924b0f9f5004b216beb6e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:42.432642 16515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53261a83ffb4e1da6f8fe5a9c043102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:42.436374 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.413277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:35651 (local address 127.0.0.1:59816)
0929 01:08:42.426812 (+ 13535us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:42.426836 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:42.426985 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:42.429147 (+ 2162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:42.429153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:42.429181 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:42.429519 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:42.429533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.431459 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.431478 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:42.433482 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:42.433497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.435592 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.435598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:42.435769 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.435804 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 01:08:42.435894 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 01:08:42.436031 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13363,"thread_start_us":13185,"threads_started":1}
I20260929 01:08:42.437146 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.413095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.5.198.195:35651)
0929 01:08:42.428268 (+ 15173us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:42.428274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:42.428302 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:08:42.428380 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:42.428385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:42.428667 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:42.428881 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:42.429718 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.431278 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.433680 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.435074 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.435209 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.435450 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:42.436654 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0929 01:08:42.436824 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15010,"thread_start_us":14873,"threads_started":1}
I20260929 01:08:42.441184 16648 tablet_service.cc:1467] Processing CreateTablet for tablet e53261a83ffb4e1da6f8fe5a9c043102 (DEFAULT_TABLE table=test-table [id=7d83d80edf924b0f9f5004b216beb6e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:08:42.442816 16648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53261a83ffb4e1da6f8fe5a9c043102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:08:42.444368 16731 tablet_bootstrap.cc:492] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b: Bootstrap starting.
I20260929 01:08:42.448328 16732 tablet_bootstrap.cc:492] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3: Bootstrap starting.
I20260929 01:08:42.448583 16731 tablet_bootstrap.cc:654] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:42.449913 16731 log.cc:824] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:08:42.452593 16732 tablet_bootstrap.cc:654] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:42.453163 16731 tablet_bootstrap.cc:492] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b: No bootstrap required, opened a new log
I20260929 01:08:42.453437 16731 ts_tablet_manager.cc:1397] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 01:08:42.453655 16732 log.cc:824] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3: Log is configured to *not* fsync() on all Append() calls
I20260929 01:08:42.456125 16732 tablet_bootstrap.cc:492] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3: No bootstrap required, opened a new log
I20260929 01:08:42.457100 16735 tablet_bootstrap.cc:492] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84: Bootstrap starting.
I20260929 01:08:42.456375 16732 ts_tablet_manager.cc:1397] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 01:08:42.460908 16735 tablet_bootstrap.cc:654] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84: Neither blocks nor log segments found. Creating new log.
I20260929 01:08:42.461772 16731 raft_consensus.cc:348] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b815f61774d989cb32539da227ae3" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46259 } } peers { permanent_uuid: "3d5e220f38624321861c416a0ca50a84" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35651 } } peers { permanent_uuid: "23ad6e9449a348e282e0fc342991d50b" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45225 } }
I20260929 01:08:42.462133 16731 raft_consensus.cc:374] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:42.462162 16735 log.cc:824] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84: Log is configured to *not* fsync() on all Append() calls
I20260929 01:08:42.462245 16731 raft_consensus.cc:729] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ad6e9449a348e282e0fc342991d50b, State: Initialized, Role: FOLLOWER
I20260929 01:08:42.462872 16731 consensus_queue.cc:260] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b815f61774d989cb32539da227ae3" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46259 } } peers { permanent_uuid: "3d5e220f38624321861c416a0ca50a84" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35651 } } peers { permanent_uuid: "23ad6e9449a348e282e0fc342991d50b" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45225 } }
I20260929 01:08:42.464600 16731 ts_tablet_manager.cc:1428] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260929 01:08:42.466939 16735 tablet_bootstrap.cc:492] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84: No bootstrap required, opened a new log
I20260929 01:08:42.467206 16735 ts_tablet_manager.cc:1397] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260929 01:08:42.469426 16732 raft_consensus.cc:348] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b815f61774d989cb32539da227ae3" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46259 } } peers { permanent_uuid: "3d5e220f38624321861c416a0ca50a84" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35651 } } peers { permanent_uuid: "23ad6e9449a348e282e0fc342991d50b" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45225 } }
I20260929 01:08:42.469890 16732 raft_consensus.cc:374] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:42.470037 16732 raft_consensus.cc:729] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e80b815f61774d989cb32539da227ae3, State: Initialized, Role: FOLLOWER
I20260929 01:08:42.470624 16732 consensus_queue.cc:260] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b815f61774d989cb32539da227ae3" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46259 } } peers { permanent_uuid: "3d5e220f38624321861c416a0ca50a84" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35651 } } peers { permanent_uuid: "23ad6e9449a348e282e0fc342991d50b" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45225 } }
I20260929 01:08:42.471639 16580 heartbeater.cc:503] Master 127.5.198.254:33123 was elected leader, sending a full tablet report...
I20260929 01:08:42.472270 16732 ts_tablet_manager.cc:1428] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3: Time spent starting tablet: real 0.015s user 0.011s sys 0.001s
I20260929 01:08:42.475984 16735 raft_consensus.cc:348] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b815f61774d989cb32539da227ae3" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46259 } } peers { permanent_uuid: "3d5e220f38624321861c416a0ca50a84" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35651 } } peers { permanent_uuid: "23ad6e9449a348e282e0fc342991d50b" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45225 } }
I20260929 01:08:42.476349 16735 raft_consensus.cc:374] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:08:42.476450 16735 raft_consensus.cc:729] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5e220f38624321861c416a0ca50a84, State: Initialized, Role: FOLLOWER
I20260929 01:08:42.477105 16735 consensus_queue.cc:260] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b815f61774d989cb32539da227ae3" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46259 } } peers { permanent_uuid: "3d5e220f38624321861c416a0ca50a84" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35651 } } peers { permanent_uuid: "23ad6e9449a348e282e0fc342991d50b" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45225 } }
I20260929 01:08:42.478093 16713 heartbeater.cc:503] Master 127.5.198.254:33123 was elected leader, sending a full tablet report...
I20260929 01:08:42.478744 16735 ts_tablet_manager.cc:1428] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260929 01:08:42.515316 16737 raft_consensus.cc:482] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:08:42.515627 16737 raft_consensus.cc:504] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b815f61774d989cb32539da227ae3" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46259 } } peers { permanent_uuid: "3d5e220f38624321861c416a0ca50a84" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35651 } } peers { permanent_uuid: "23ad6e9449a348e282e0fc342991d50b" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45225 } }
I20260929 01:08:42.517468 16737 leader_election.cc:288] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e80b815f61774d989cb32539da227ae3 (127.5.198.194:46259), 3d5e220f38624321861c416a0ca50a84 (127.5.198.195:35651)
I20260929 01:08:42.524601 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.517936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:46707 (local address 127.5.198.194:46259)
0929 01:08:42.518165 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:42.518172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:42.518599 (+ 427us) server_negotiation.cc:407] Connection header received
0929 01:08:42.518835 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:42.518841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:42.518916 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:42.518981 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:42.519800 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.520740 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.522734 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.523609 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.523727 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.523783 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:42.524162 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 01:08:42.524319 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:08:42.524688 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.518837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:33207 (local address 127.5.198.195:35651)
0929 01:08:42.519055 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:42.519061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:42.519304 (+ 243us) server_negotiation.cc:407] Connection header received
0929 01:08:42.519454 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:42.519458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:42.519535 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:42.519636 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:08:42.520289 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.521195 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.523010 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.523807 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.523959 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.524035 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:42.524291 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 01:08:42.524424 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:08:42.524678 16740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.517839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:46259 (local address 127.5.198.193:46707)
0929 01:08:42.518458 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:42.518484 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:42.518650 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:42.519185 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:42.519192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:42.519213 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:42.519633 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:42.519645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.520856 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.520862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:42.522585 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:42.522596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.523808 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.523814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:42.523992 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.524028 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 01:08:42.524117 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 01:08:42.524207 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":67,"thread_start_us":355,"threads_started":1}
I20260929 01:08:42.524807 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.518575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:35651 (local address 127.5.198.193:33207)
0929 01:08:42.519200 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 01:08:42.519217 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:08:42.519336 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:08:42.519656 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:08:42.519659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:08:42.519677 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:08:42.520119 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:42.520131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.521320 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.521324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:42.522856 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:08:42.522870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.523993 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.523997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:08:42.524135 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.524165 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 01:08:42.524253 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 01:08:42.524371 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":1,"thread_start_us":333,"threads_started":1}
I20260929 01:08:42.525871 16668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53261a83ffb4e1da6f8fe5a9c043102" candidate_uuid: "23ad6e9449a348e282e0fc342991d50b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5e220f38624321861c416a0ca50a84" is_pre_election: true
I20260929 01:08:42.525859 16535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53261a83ffb4e1da6f8fe5a9c043102" candidate_uuid: "23ad6e9449a348e282e0fc342991d50b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e80b815f61774d989cb32539da227ae3" is_pre_election: true
I20260929 01:08:42.526448 16535 raft_consensus.cc:2454] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ad6e9449a348e282e0fc342991d50b in term 0.
I20260929 01:08:42.526446 16668 raft_consensus.cc:2454] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ad6e9449a348e282e0fc342991d50b in term 0.
I20260929 01:08:42.527124 16335 leader_election.cc:302] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ad6e9449a348e282e0fc342991d50b, e80b815f61774d989cb32539da227ae3; no voters:
I20260929 01:08:42.527444 16737 raft_consensus.cc:2781] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:08:42.527601 16737 raft_consensus.cc:482] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:08:42.527704 16737 raft_consensus.cc:3037] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:42.529918 16737 raft_consensus.cc:504] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b815f61774d989cb32539da227ae3" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46259 } } peers { permanent_uuid: "3d5e220f38624321861c416a0ca50a84" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35651 } } peers { permanent_uuid: "23ad6e9449a348e282e0fc342991d50b" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45225 } }
I20260929 01:08:42.530467 16737 leader_election.cc:288] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [CANDIDATE]: Term 1 election: Requested vote from peers e80b815f61774d989cb32539da227ae3 (127.5.198.194:46259), 3d5e220f38624321861c416a0ca50a84 (127.5.198.195:35651)
I20260929 01:08:42.530974 16535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53261a83ffb4e1da6f8fe5a9c043102" candidate_uuid: "23ad6e9449a348e282e0fc342991d50b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e80b815f61774d989cb32539da227ae3"
I20260929 01:08:42.531028 16668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53261a83ffb4e1da6f8fe5a9c043102" candidate_uuid: "23ad6e9449a348e282e0fc342991d50b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d5e220f38624321861c416a0ca50a84"
I20260929 01:08:42.531179 16535 raft_consensus.cc:3037] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:42.531265 16668 raft_consensus.cc:3037] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:08:42.533766 16535 raft_consensus.cc:2454] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ad6e9449a348e282e0fc342991d50b in term 1.
I20260929 01:08:42.533972 16668 raft_consensus.cc:2454] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ad6e9449a348e282e0fc342991d50b in term 1.
I20260929 01:08:42.534148 16335 leader_election.cc:302] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ad6e9449a348e282e0fc342991d50b, e80b815f61774d989cb32539da227ae3; no voters:
I20260929 01:08:42.534420 16737 raft_consensus.cc:2781] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:08:42.534849 16737 raft_consensus.cc:686] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [term 1 LEADER]: Becoming Leader. State: Replica: 23ad6e9449a348e282e0fc342991d50b, State: Running, Role: LEADER
I20260929 01:08:42.535305 16737 consensus_queue.cc:237] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b815f61774d989cb32539da227ae3" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46259 } } peers { permanent_uuid: "3d5e220f38624321861c416a0ca50a84" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35651 } } peers { permanent_uuid: "23ad6e9449a348e282e0fc342991d50b" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45225 } }
I20260929 01:08:42.539794 16257 catalog_manager.cc:5495] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b reported cstate change: term changed from 0 to 1, leader changed from <none> to 23ad6e9449a348e282e0fc342991d50b (127.5.198.193). New cstate: current_term: 1 leader_uuid: "23ad6e9449a348e282e0fc342991d50b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b815f61774d989cb32539da227ae3" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d5e220f38624321861c416a0ca50a84" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ad6e9449a348e282e0fc342991d50b" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45225 } health_report { overall_health: HEALTHY } } }
I20260929 01:08:42.569250 16727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.564070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36322 (local address 127.5.198.193:45225)
0929 01:08:42.564282 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:42.564289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:42.564310 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:08:42.564409 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:42.564413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:42.564541 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:42.564679 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:08:42.565079 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.565790 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.567069 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.567502 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.567625 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.568115 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:08:42.568674 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:08:42.568820 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:42.568915 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 01:08:42.568994 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:08:42.577376 16535 raft_consensus.cc:1261] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3 [term 1 FOLLOWER]: Refusing update from remote peer 23ad6e9449a348e282e0fc342991d50b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:08:42.577417 16668 raft_consensus.cc:1261] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84 [term 1 FOLLOWER]: Refusing update from remote peer 23ad6e9449a348e282e0fc342991d50b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 01:08:42.577639 16447 tablet.cc:2369] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:08:42.578473 16742 consensus_queue.cc:1035] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d5e220f38624321861c416a0ca50a84" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 35651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:08:42.578824 16737 consensus_queue.cc:1035] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e80b815f61774d989cb32539da227ae3" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 01:08:42.587162 16714 tablet.cc:2369] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:08:42.590445 16745 mvcc.cc:205] Tried to move back new op lower bound from 7334478326066720768 to 7334478325908013056. Current Snapshot: MvccSnapshot[applied={T|T < 7334478326066720768}]
W20260929 01:08:42.591765 16581 tablet.cc:2369] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:08:42.592144 16746 mvcc.cc:205] Tried to move back new op lower bound from 7334478326066720768 to 7334478325908013056. Current Snapshot: MvccSnapshot[applied={T|T < 7334478326066720768}]
I20260929 01:08:42.597657 16747 mvcc.cc:205] Tried to move back new op lower bound from 7334478326066720768 to 7334478325908013056. Current Snapshot: MvccSnapshot[applied={T|T < 7334478326066720768}]
I20260929 01:08:42.605790 16256 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:08:42.606091 16256 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:08:42.609500 16256 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f869c8e746d4801ae2a9a439156efcc: Sending DeleteTablet for 3 replicas of tablet e53261a83ffb4e1da6f8fe5a9c043102
I20260929 01:08:42.610663 16381 tablet_service.cc:1514] Processing DeleteTablet for tablet e53261a83ffb4e1da6f8fe5a9c043102 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:42 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:36306
I20260929 01:08:42.610657 16648 tablet_service.cc:1514] Processing DeleteTablet for tablet e53261a83ffb4e1da6f8fe5a9c043102 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:42 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:59816
I20260929 01:08:42.610734 16515 tablet_service.cc:1514] Processing DeleteTablet for tablet e53261a83ffb4e1da6f8fe5a9c043102 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:08:42 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:55852
I20260929 01:08:42.612874 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 16318
I20260929 01:08:42.616956 16756 tablet_replica.cc:330] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3: stopping tablet replica
I20260929 01:08:42.617319 16756 raft_consensus.cc:2229] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:08:42.617743 16756 raft_consensus.cc:2258] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:42.619856 16756 ts_tablet_manager.cc:1905] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:42.626878 16756 ts_tablet_manager.cc:1918] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:08:42.627108 16756 log.cc:1196] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/e53261a83ffb4e1da6f8fe5a9c043102
I20260929 01:08:42.627732 16756 ts_tablet_manager.cc:1939] T e53261a83ffb4e1da6f8fe5a9c043102 P e80b815f61774d989cb32539da227ae3: Deleting consensus metadata
I20260929 01:08:42.630852 16242 catalog_manager.cc:4841] TS e80b815f61774d989cb32539da227ae3 (127.5.198.194:46259): tablet e53261a83ffb4e1da6f8fe5a9c043102 (table test-table [id=7d83d80edf924b0f9f5004b216beb6e6]) successfully deleted
I20260929 01:08:42.633713 16754 tablet_replica.cc:330] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b: stopping tablet replica
I20260929 01:08:42.634240 16754 raft_consensus.cc:2229] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:08:42.634994 16754 raft_consensus.cc:2258] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:42.636061 16755 tablet_replica.cc:330] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84: stopping tablet replica
I20260929 01:08:42.636560 16755 raft_consensus.cc:2229] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:08:42.637117 16755 raft_consensus.cc:2258] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:08:42.637857 16754 ts_tablet_manager.cc:1905] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:08:42.639038 16755 ts_tablet_manager.cc:1905] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 01:08:42 dist-test-slave-41cl krb5kdc[16212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:08:42.645627 16754 ts_tablet_manager.cc:1918] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:08:42.645803 16755 ts_tablet_manager.cc:1918] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:08:42.645902 16754 log.cc:1196] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/e53261a83ffb4e1da6f8fe5a9c043102
I20260929 01:08:42.646000 16755 log.cc:1196] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/e53261a83ffb4e1da6f8fe5a9c043102
I20260929 01:08:42.646518 16754 ts_tablet_manager.cc:1939] T e53261a83ffb4e1da6f8fe5a9c043102 P 23ad6e9449a348e282e0fc342991d50b: Deleting consensus metadata
I20260929 01:08:42.646551 16755 ts_tablet_manager.cc:1939] T e53261a83ffb4e1da6f8fe5a9c043102 P 3d5e220f38624321861c416a0ca50a84: Deleting consensus metadata
I20260929 01:08:42.648324 16243 catalog_manager.cc:4841] TS 23ad6e9449a348e282e0fc342991d50b (127.5.198.193:45225): tablet e53261a83ffb4e1da6f8fe5a9c043102 (table test-table [id=7d83d80edf924b0f9f5004b216beb6e6]) successfully deleted
I20260929 01:08:42.648825 16242 catalog_manager.cc:4841] TS 3d5e220f38624321861c416a0ca50a84 (127.5.198.195:35651): tablet e53261a83ffb4e1da6f8fe5a9c043102 (table test-table [id=7d83d80edf924b0f9f5004b216beb6e6]) successfully deleted
I20260929 01:08:42.651831 16727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.614692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.5.198.193:45225)
0929 01:08:42.614909 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:42.614918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:42.633030 (+ 18112us) server_negotiation.cc:407] Connection header received
0929 01:08:42.633228 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:42.633241 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:42.633422 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:42.633570 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:42.635761 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.636942 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.638712 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.639446 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.639551 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.644851 (+ 5300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:42.644864 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:42.644882 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:42.644920 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:42.649192 (+ 4272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:42.649998 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:42.650003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:42.650006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:42.650126 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:42.650519 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:42.650522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:42.650524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:42.650834 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:42.651069 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:42.651362 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 01:08:42.651521 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:08:42.652535 16441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:42.749804 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 16318
I20260929 01:08:42.768085 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 16452
Sep 29 01:08:42 dist-test-slave-41cl krb5kdc[16212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:08:42.782828 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.768805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.5.198.194:46259)
0929 01:08:42.768975 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:42.768982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:42.769025 (+ 43us) server_negotiation.cc:407] Connection header received
0929 01:08:42.769230 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:42.769234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:42.769295 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:42.769408 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:42.770594 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.771306 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.773071 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.773581 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.773712 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.777669 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:42.777682 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:42.777697 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:42.777728 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:42.780360 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:42.781184 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:42.781191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:42.781193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:42.781304 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:42.781622 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:42.781625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:42.781627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:42.781960 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:42.782224 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:42.782436 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 01:08:42.782582 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:08:42.783510 16575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:42.871357 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 16452
I20260929 01:08:42.888737 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 16585
Sep 29 01:08:42 dist-test-slave-41cl krb5kdc[16212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:08:42.903726 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:08:42.889544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.5.198.195:35651)
0929 01:08:42.889752 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 01:08:42.889757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:08:42.889827 (+ 70us) server_negotiation.cc:407] Connection header received
0929 01:08:42.890001 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:08:42.890005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:08:42.890068 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:08:42.890153 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:08:42.891413 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.892174 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.893725 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:08:42.894105 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:08:42.894200 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:08:42.898422 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:08:42.898436 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:08:42.898452 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:08:42.898506 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:08:42.901180 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:42.901925 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:42.901932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:42.901935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:42.902066 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:08:42.902383 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:08:42.902386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:08:42.902388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:08:42.902716 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:08:42.902984 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 01:08:42.903194 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 01:08:42.903359 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 01:08:42.904356 16708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:42.991822 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 16585
I20260929 01:08:43.008323 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 16225
I20260929 01:08:43.009603 16287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:08:43.116894 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 16225
2026-09-29T01:08:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4138 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:08:43 dist-test-slave-41cl krb5kdc[16764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:08:43 dist-test-slave-41cl krb5kdc[16764](info): set up 2 sockets
Sep 29 01:08:43 dist-test-slave-41cl krb5kdc[16764](info): commencing operation
krb5kdc: starting...
W20260929 01:08:44.225648 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:08:44 dist-test-slave-41cl krb5kdc[16764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:08:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:08:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.198.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.198.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.198.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/rangeradmin_127.5.198.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.198.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/rangeradmin_127.5.198.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.198.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.198.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.198.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/rangerlookup_127.5.198.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.198.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/rangerlookup_127.5.198.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.198.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.198.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.198.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/HTTP_127.5.198.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.198.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/HTTP_127.5.198.212@KRBTEST.COM.keytab.
I20260929 01:08:44.555770 5915 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskd3rvDY/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/postgres -l logfile start
2026-09-29 01:08:47.197 UTC [16794] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 01:08:47.197 UTC [16794] LOG: listening on IPv4 address "127.5.198.212", port 33793
2026-09-29 01:08:47.201 UTC [16794] LOG: listening on Unix socket "/tmp/.s.PGSQL.33793"
2026-09-29 01:08:47.223 UTC [16797] LOG: database system was shut down at 2026-09-29 01:08:45 UTC
2026-09-29 01:08:47.226 UTC [16794] LOG: database system is ready to accept connections
I20260929 01:08:48.197019 5915 mini_postgres.cc:96] Postgres bound to 33793
2026-09-29 01:08:48.213 UTC [16805] FATAL: database "slave" does not exist
127.5.198.212:33793 - accepting connections
I20260929 01:08:48.214695 5915 mini_ranger.cc:161] Starting Ranger...
I20260929 01:08:48.239302 5915 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 01:08:48.771733 5915 mini_ranger.cc:87] Created ranger Postgres database
I20260929 01:08:48.771888 5915 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-admin
2026-09-29 01:08:49,063 [I] DB FLAVOR :POSTGRES
2026-09-29 01:08:49,064 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 01:08:49,064 [I] Checking connection..
2026-09-29 01:08:49,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 01:08:49,326 [I] Checking connection passed.
2026-09-29 01:08:49,326 [I] --------- Verifying version history table ---------
2026-09-29 01:08:49,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:08:49,575 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 01:08:49,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:08:49,806 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 01:08:49,806 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 01:08:49,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 01:08:50.060 UTC [16869] WARNING: there is no transaction in progress
2026-09-29 01:08:50,066 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 01:08:50,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:08:50,314 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 01:08:50,315 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 01:08:50,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 01:08:50,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 01:08:50,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-41cl.c.gcp-upstream.internal','N') ;"
2026-09-29 01:08:51,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 01:08:51,255 [I] Table x_portal_user does not exist in database ranger
2026-09-29 01:08:51,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 01:08:51,502 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 01:08:51,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 01:08:51,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 01:08:51,969 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 01:08:51,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 01:08:52.972 UTC [16977] WARNING: there is no transaction in progress
2026-09-29 01:08:52.989 UTC [16977] WARNING: there is no transaction in progress
2026-09-29 01:08:53.007 UTC [16977] WARNING: there is no transaction in progress
2026-09-29 01:08:53.024 UTC [16977] WARNING: there is no transaction in progress
2026-09-29 01:08:53.041 UTC [16977] WARNING: there is no transaction in progress
2026-09-29 01:08:53.062 UTC [16977] WARNING: there is no transaction in progress
2026-09-29 01:08:53.124 UTC [16977] WARNING: there is no transaction in progress
2026-09-29 01:08:53.131 UTC [16977] WARNING: there is no transaction in progress
2026-09-29 01:08:53.146 UTC [16977] WARNING: there is no transaction in progress
2026-09-29 01:08:53.156 UTC [16977] WARNING: there is no transaction in progress
2026-09-29 01:08:53.167 UTC [16977] WARNING: there is no transaction in progress
2026-09-29 01:08:53.183 UTC [16977] WARNING: there is no transaction in progress
2026-09-29 01:08:53.193 UTC [16977] WARNING: there is no transaction in progress
2026-09-29 01:08:53.200 UTC [16977] WARNING: there is no transaction in progress
2026-09-29 01:08:53.207 UTC [16977] WARNING: there is no transaction in progress
2026-09-29 01:08:53.215 UTC [16977] WARNING: there is no transaction in progress
2026-09-29 01:08:53,909 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 01:08:53,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 01:08:54,142 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 01:08:54,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 01:08:54,393 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 01:08:54,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 01:08:54,626 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 01:08:54,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 01:08:54,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 01:08:55,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-41cl.c.gcp-upstream.internal';"
2026-09-29 01:08:55,364 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 01:08:55,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 01:08:55,606 [I] DB_PATCHES have already been applied
I20260929 01:08:55.612640 5915 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-admin:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskd3rvDY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 01:08:55.612764 5915 mini_ranger.cc:193] Using host: 127.5.198.212
I20260929 01:08:55.628453 5915 mini_ranger.cc:227] Ranger admin URL: http://127.5.198.212:33597
Sep 29, 2026 1:08:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 1:08:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 1:08:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 1:08:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 1:08:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 1:08:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 1:08:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 01:08:56.790658 5915 mini_ranger.cc:160] Time spent starting Ranger: real 8.576s user 0.009s sys 0.031s
Sep 29, 2026 1:09:16 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 1:09:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 1:09:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 1:09:16 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 1:09:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 1:09:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 1:09:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 1:09:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 1:09:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 1:09:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 1:09:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 1:09:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 1:09:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 1:09:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 1:09:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 1:09:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 1:09:16 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 1:09:17 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 01:09:18.308655 5915 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.198.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.198.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.198.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/rangerkms_127.5.198.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.198.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/rangerkms_127.5.198.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.198.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.198.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.198.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/HTTP_127.5.198.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.198.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/HTTP_127.5.198.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 01:09:18 dist-test-slave-41cl krb5kdc[16764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644158, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 01:09:18.482713 5915 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 01:09:18.503141 5915 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 01:09:19.450551 5915 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 01:09:19.450717 5915 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-kms
I20260929 01:09:19.450853 5915 mini_ranger_kms.cc:227] Using postgres at 127.5.198.212:33793
2026-09-29 01:09:19,868 [I] DB FLAVOR :POSTGRES
2026-09-29 01:09:19,868 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 01:09:19,868 [I] Checking connection
2026-09-29 01:09:19,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 01:09:20,130 [I] connection success
2026-09-29 01:09:20,130 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 01:09:20,130 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 01:09:20,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 01:09:20,372 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 01:09:20,372 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 01:09:20,372 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 01:09:20,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:33793/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 01:09:20,622 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 01:09:21.067703 5915 mini_ranger_kms.cc:310] Created kms service
I20260929 01:09:21.269017 5915 mini_ranger_kms.cc:326] Created kudu user
I20260929 01:09:21.318413 5915 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 01:09:21 dist-test-slave-41cl krb5kdc[16764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644158, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
I20260929 01:09:21.723709 5915 mini_ranger_kms.cc:384] Added ranger policy
I20260929 01:09:21.724020 5915 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-kms:/tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 01:09:21.724066 5915 mini_ranger_kms.cc:240] Using host: 127.5.198.212
I20260929 01:09:21.741614 5915 mini_ranger_kms.cc:276] Ranger KMS PID: 17157
I20260929 01:09:21.741808 5915 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.198.212:35083
Sep 29, 2026 1:09:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 1:09:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 1:09:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 1:09:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 1:09:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 1:09:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 01:09:24.028434 5915 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.546s user 0.110s sys 0.321s
I20260929 01:09:24.028731 5915 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 01:09:24.028936 5915 mini_ranger_kms.cc:401] 127.5.198.212:35083/kms/v1/keys
Sep 29 01:09:24 dist-test-slave-41cl krb5kdc[16764](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790644164, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.198.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:09:24 dist-test-slave-41cl krb5kdc[16764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644164, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.198.212@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
Sep 29 01:09:24 dist-test-slave-41cl krb5kdc[16764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644164, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.198.212@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
Sep 29 01:09:26 dist-test-slave-41cl krb5kdc[16764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:09:27.009294 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:33849
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:33849
--ranger_config_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.198.212:35083/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:09:27.338528 17208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:09:27.338796 17208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:09:27.338963 17208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:09:27.345851 17208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:09:27.345952 17208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:09:27.346038 17208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:09:27.346120 17208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:09:27.346230 17208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:09:27.356132 17208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.198.212:35083/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:33849
--ranger_config_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:33849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:09:27.357442 17208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:09:27.358724 17208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:09:27.371088 17213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:09:27.370983 17216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:09:27.371131 17214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:09:27.371570 17208 server_base.cc:1034] running on GCE node
I20260929 01:09:27.372306 17208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:09:27.373713 17208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:09:27.374950 17208 hybrid_clock.cc:648] HybridClock initialized: now 1790644167374899 us; error 63 us; skew 500 ppm
I20260929 01:09:27.375298 17208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:09:27 dist-test-slave-41cl krb5kdc[16764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644167, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:09:27.378459 17208 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:09:27.379838 17208 webserver.cc:460] Webserver started at http://127.5.198.254:40067/ using document root <none> and password file <none>
I20260929 01:09:27.380409 17208 fs_manager.cc:359] Metadata directory not provided
I20260929 01:09:27.380563 17208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:09:27.380810 17208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:09:27 dist-test-slave-41cl krb5kdc[16764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644167, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
I20260929 01:09:27.449019 17208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "0357ba9502134d17847b40d44a05841e"
format_stamp: "Formatted at 2026-09-29 01:09:27 on dist-test-slave-41cl"
server_key: "117c05caf6a5cae8f626aff445d0e1c8"
server_key_iv: "7a8e456683a32b1aeab49d639e33b9a8"
server_key_version: "kuduclusterkey@0"
I20260929 01:09:27.449870 17208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "0357ba9502134d17847b40d44a05841e"
format_stamp: "Formatted at 2026-09-29 01:09:27 on dist-test-slave-41cl"
server_key: "117c05caf6a5cae8f626aff445d0e1c8"
server_key_iv: "7a8e456683a32b1aeab49d639e33b9a8"
server_key_version: "kuduclusterkey@0"
I20260929 01:09:27.455360 17208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:09:27.475651 17223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:09:27.477205 17208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:09:27.477386 17208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "0357ba9502134d17847b40d44a05841e"
format_stamp: "Formatted at 2026-09-29 01:09:27 on dist-test-slave-41cl"
server_key: "117c05caf6a5cae8f626aff445d0e1c8"
server_key_iv: "7a8e456683a32b1aeab49d639e33b9a8"
server_key_version: "kuduclusterkey@0"
I20260929 01:09:27.477630 17208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:09:27.496765 17208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:09:27.500582 17208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:09:27.500878 17208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:09:27.520098 17208 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:33849
I20260929 01:09:27.520121 17274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:33849 every 8 connection(s)
I20260929 01:09:27.521520 17208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:09:27.524276 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 17208
I20260929 01:09:27.524502 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:09:27.525100 5915 external_mini_cluster.cc:1428] Setting key 3b562fe0dc8fe0c2dc0c85de6ffacbe2
I20260929 01:09:27.525990 17275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 01:09:27 dist-test-slave-41cl krb5kdc[16764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:09:27.541141 17275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e: Bootstrap starting.
I20260929 01:09:27.545560 17275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e: Neither blocks nor log segments found. Creating new log.
I20260929 01:09:27.546980 17275 log.cc:824] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:09:27.549290 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:27.526984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39748 (local address 127.5.198.254:33849)
0929 01:09:27.527692 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0929 01:09:27.527703 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:09:27.527742 (+ 39us) server_negotiation.cc:407] Connection header received
0929 01:09:27.529358 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:09:27.529378 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:09:27.529749 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:09:27.530079 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:09:27.531814 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:27.533268 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:27.535097 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:27.535744 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:27.535876 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:27.541014 (+ 5138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:09:27.541026 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:09:27.541044 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:09:27.541097 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:09:27.545037 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:09:27.545691 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:09:27.545700 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:09:27.545708 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:09:27.545930 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:09:27.546298 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:09:27.546303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:09:27.546306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:09:27.547024 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:09:27.547310 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 01:09:27.547930 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 01:09:27.548333 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":328,"threads_started":1}
I20260929 01:09:27.550132 17275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e: No bootstrap required, opened a new log
I20260929 01:09:27.557845 17275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0357ba9502134d17847b40d44a05841e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33849 } }
I20260929 01:09:27.558194 17275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:09:27.558332 17275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0357ba9502134d17847b40d44a05841e, State: Initialized, Role: FOLLOWER
I20260929 01:09:27.558940 17275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0357ba9502134d17847b40d44a05841e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33849 } }
I20260929 01:09:27.559175 17275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:09:27.559342 17275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:09:27.559531 17275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:09:27.561798 17275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0357ba9502134d17847b40d44a05841e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33849 } }
I20260929 01:09:27.562327 17275 leader_election.cc:302] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0357ba9502134d17847b40d44a05841e; no voters:
I20260929 01:09:27.562907 17275 leader_election.cc:288] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:09:27.563056 17280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:09:27.563766 17280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [term 1 LEADER]: Becoming Leader. State: Replica: 0357ba9502134d17847b40d44a05841e, State: Running, Role: LEADER
I20260929 01:09:27.565038 17275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:09:27.564967 17280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0357ba9502134d17847b40d44a05841e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33849 } }
I20260929 01:09:27.568267 17281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0357ba9502134d17847b40d44a05841e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0357ba9502134d17847b40d44a05841e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33849 } } }
I20260929 01:09:27.568681 17281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [sys.catalog]: This master's current role is: LEADER
I20260929 01:09:27.568521 17282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0357ba9502134d17847b40d44a05841e. Latest consensus state: current_term: 1 leader_uuid: "0357ba9502134d17847b40d44a05841e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0357ba9502134d17847b40d44a05841e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33849 } } }
I20260929 01:09:27.569053 17282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e [sys.catalog]: This master's current role is: LEADER
I20260929 01:09:27.573199 17288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:09:27.579358 17288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:09:27.580152 17275 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 01:09:27.598124 17288 catalog_manager.cc:1347] Generated new cluster ID: 0417a5e9c0284389b32053bd7f91edb9
I20260929 01:09:27.598318 17288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:09:27.622956 17288 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:09:27.624604 17288 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:09:27.631458 17288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e: Generated new TSK 0
I20260929 01:09:27.632299 17288 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:09:28.450641 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.198.212:35083/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33849
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:09:28 dist-test-slave-41cl krb5kdc[16764](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790644168, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 01:09:28.880851 17320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:09:28.881165 17320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:09:28.881327 17320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:09:28.889463 17320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:09:28.889564 17320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:09:28.889722 17320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:09:28.900285 17320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.198.212:35083/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:09:28.901798 17320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:09:28.903337 17320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:09:28.920944 17330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:09:28.922094 17331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:09:28.937170 17333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:09:28.938706 17320 server_base.cc:1034] running on GCE node
I20260929 01:09:28.939605 17320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:09:28.940965 17320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:09:28.942201 17320 hybrid_clock.cc:648] HybridClock initialized: now 1790644168942132 us; error 49 us; skew 500 ppm
I20260929 01:09:28.942617 17320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:09:28 dist-test-slave-41cl krb5kdc[16764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644168, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:09:28.953145 17320 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:09:28.962775 17320 webserver.cc:460] Webserver started at http://127.5.198.193:41957/ using document root <none> and password file <none>
I20260929 01:09:28.963758 17320 fs_manager.cc:359] Metadata directory not provided
I20260929 01:09:28.963904 17320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:09:28.964265 17320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:09:28 dist-test-slave-41cl krb5kdc[16764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644168, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
I20260929 01:09:28.984077 17320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "a1c11acaf3bb4e4daa5de70062b3dbc1"
format_stamp: "Formatted at 2026-09-29 01:09:28 on dist-test-slave-41cl"
server_key: "307c76f74d896b3692959e38dd6283ff"
server_key_iv: "db29dafac758220bf49fefa94ea77558"
server_key_version: "kuduclusterkey@0"
I20260929 01:09:28.985320 17320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "a1c11acaf3bb4e4daa5de70062b3dbc1"
format_stamp: "Formatted at 2026-09-29 01:09:28 on dist-test-slave-41cl"
server_key: "307c76f74d896b3692959e38dd6283ff"
server_key_iv: "db29dafac758220bf49fefa94ea77558"
server_key_version: "kuduclusterkey@0"
I20260929 01:09:28.998580 17320 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.007s sys 0.004s
I20260929 01:09:29.027128 17340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:09:29.029901 17320 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.001s
I20260929 01:09:29.030155 17320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "a1c11acaf3bb4e4daa5de70062b3dbc1"
format_stamp: "Formatted at 2026-09-29 01:09:28 on dist-test-slave-41cl"
server_key: "307c76f74d896b3692959e38dd6283ff"
server_key_iv: "db29dafac758220bf49fefa94ea77558"
server_key_version: "kuduclusterkey@0"
I20260929 01:09:29.030509 17320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:09:29.054241 17320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:09:29.059162 17320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:09:29.059494 17320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:09:29.060739 17320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:09:29.062434 17320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:09:29.062525 17320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:09:29.062647 17320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:09:29.062704 17320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:09:29.130040 17320 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:45157
I20260929 01:09:29.130080 17452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:45157 every 8 connection(s)
I20260929 01:09:29.131920 17320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:09:29.139102 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 17320
I20260929 01:09:29.139294 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:09:29.140113 5915 external_mini_cluster.cc:1428] Setting key 1a565cdd67a3411cb8bfb412f748a9d5
Sep 29 01:09:29 dist-test-slave-41cl krb5kdc[16764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644168, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:09:29.159816 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:29.136001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:43565 (local address 127.5.198.254:33849)
0929 01:09:29.136507 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 01:09:29.136515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:09:29.137524 (+ 1009us) server_negotiation.cc:407] Connection header received
0929 01:09:29.140128 (+ 2604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:09:29.140134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:09:29.140217 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:09:29.140305 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:09:29.142535 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:29.143656 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:29.146070 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:29.146700 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:29.146799 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:29.151372 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:09:29.151379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:09:29.151384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:09:29.151422 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:09:29.155055 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:09:29.156432 (+ 1377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:09:29.156438 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:09:29.156442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:09:29.156633 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:09:29.157263 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:09:29.157272 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:09:29.157278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:09:29.157805 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:09:29.157949 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 01:09:29.159057 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0929 01:09:29.159311 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":217,"threads_started":1}
I20260929 01:09:29.162271 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:29.136593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33849 (local address 127.5.198.193:43565)
0929 01:09:29.137343 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0929 01:09:29.137395 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 01:09:29.139789 (+ 2394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:09:29.140868 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:09:29.140934 (+ 66us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:09:29.141543 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:09:29.142335 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:09:29.142362 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:09:29.143864 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:09:29.143870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:09:29.145908 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:09:29.145922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:09:29.147150 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:09:29.147158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:09:29.147433 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:29.148266 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:09:29.148297 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:09:29.151119 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:09:29.155220 (+ 4101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:09:29.155228 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:09:29.155245 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:09:29.156288 (+ 1043us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:09:29.156787 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:09:29.156791 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:09:29.156794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:09:29.157050 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:09:29.158048 (+ 998us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:09:29.158059 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:09:29.158618 (+ 559us) client_negotiation.cc:769] Sending connection context
0929 01:09:29.160112 (+ 1494us) client_negotiation.cc:240] Negotiation successful
0929 01:09:29.160765 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":309,"threads_started":1}
I20260929 01:09:29.164326 17453 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33849
I20260929 01:09:29.164691 17453 heartbeater.cc:464] Registering TS with master...
I20260929 01:09:29.165597 17453 heartbeater.cc:511] Master 127.5.198.254:33849 requested a full tablet report, sending...
I20260929 01:09:29.168243 17240 ts_manager.cc:194] Registered new tserver with Master: a1c11acaf3bb4e4daa5de70062b3dbc1 (127.5.198.193:45157)
I20260929 01:09:29.170500 17240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:43565
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:09:29.207528 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.198.212:35083/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33849
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:09:29 dist-test-slave-41cl krb5kdc[16764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644168, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
Sep 29 01:09:29 dist-test-slave-41cl krb5kdc[16764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644168, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
W20260929 01:09:29.681942 17459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:09:29.682432 17459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:09:29.682633 17459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:09:29.694563 17459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:09:29.694796 17459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:09:29.695050 17459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:09:29.710577 17459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.198.212:35083/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:09:29.712704 17459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:09:29.714534 17459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:09:29.742606 17467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:09:29.743196 17466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:09:29.745822 17469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:09:29.747108 17459 server_base.cc:1034] running on GCE node
I20260929 01:09:29.748021 17459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:09:29.750440 17459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:09:29.751755 17459 hybrid_clock.cc:648] HybridClock initialized: now 1790644169751651 us; error 88 us; skew 500 ppm
I20260929 01:09:29.752144 17459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:09:29 dist-test-slave-41cl krb5kdc[16764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644169, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:09:29.756744 17459 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:09:29.782994 17459 webserver.cc:460] Webserver started at http://127.5.198.194:32815/ using document root <none> and password file <none>
I20260929 01:09:29.784195 17459 fs_manager.cc:359] Metadata directory not provided
I20260929 01:09:29.784451 17459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:09:29.784963 17459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:09:29 dist-test-slave-41cl krb5kdc[16764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644169, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
I20260929 01:09:29.808858 17459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "d703b9a5e94c4ec5b153ce6874ea5448"
format_stamp: "Formatted at 2026-09-29 01:09:29 on dist-test-slave-41cl"
server_key: "12a07c5eda240599e5e023d007594652"
server_key_iv: "5c311010d09f7a506dc3ea59b0abfff8"
server_key_version: "kuduclusterkey@0"
I20260929 01:09:29.809752 17459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "d703b9a5e94c4ec5b153ce6874ea5448"
format_stamp: "Formatted at 2026-09-29 01:09:29 on dist-test-slave-41cl"
server_key: "12a07c5eda240599e5e023d007594652"
server_key_iv: "5c311010d09f7a506dc3ea59b0abfff8"
server_key_version: "kuduclusterkey@0"
I20260929 01:09:29.815536 17459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 01:09:29.849241 17476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:09:29.851557 17459 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.004s sys 0.011s
I20260929 01:09:29.855621 17459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "d703b9a5e94c4ec5b153ce6874ea5448"
format_stamp: "Formatted at 2026-09-29 01:09:29 on dist-test-slave-41cl"
server_key: "12a07c5eda240599e5e023d007594652"
server_key_iv: "5c311010d09f7a506dc3ea59b0abfff8"
server_key_version: "kuduclusterkey@0"
I20260929 01:09:29.856000 17459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:09:29.927439 17459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:09:29.932776 17459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:09:29.933105 17459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:09:29.934337 17459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:09:29.936143 17459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:09:29.936234 17459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:09:29.936393 17459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:09:29.936450 17459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:09:29.983143 17459 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:39321
I20260929 01:09:29.983180 17595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:39321 every 8 connection(s)
I20260929 01:09:29.984879 17459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:09:29.990518 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 17459
I20260929 01:09:29.990731 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:09:29.991253 5915 external_mini_cluster.cc:1428] Setting key 388a5674f00e2fb3cfca09fa2d736c78
Sep 29 01:09:30 dist-test-slave-41cl krb5kdc[16764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644169, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:09:30.015998 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:29.988334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33849 (local address 127.5.198.194:53629)
0929 01:09:29.993411 (+ 5077us) negotiation.cc:107] Waiting for socket to connect
0929 01:09:29.993458 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 01:09:29.995797 (+ 2339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:09:29.996892 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:09:29.996905 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:09:29.997488 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:09:29.998275 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:09:29.998301 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:09:29.999826 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:09:29.999832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:09:30.001815 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:09:30.001829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.002870 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.002878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:09:30.003163 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:30.004010 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:09:30.004040 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:09:30.006849 (+ 2809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:09:30.010924 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:09:30.010933 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:09:30.010951 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:09:30.011526 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:09:30.011971 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:09:30.011976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:09:30.011979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:09:30.012214 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:09:30.013333 (+ 1119us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:09:30.013341 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:09:30.013800 (+ 459us) client_negotiation.cc:769] Sending connection context
0929 01:09:30.014090 (+ 290us) client_negotiation.cc:240] Negotiation successful
0929 01:09:30.014615 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4779,"thread_start_us":4510,"threads_started":1}
I20260929 01:09:30.017863 17598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:29.987532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:53629 (local address 127.5.198.254:33849)
0929 01:09:29.991059 (+ 3527us) server_negotiation.cc:206] Beginning negotiation
0929 01:09:29.991066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:09:29.996160 (+ 5094us) server_negotiation.cc:407] Connection header received
0929 01:09:29.996241 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:09:29.996247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:09:29.996328 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:09:29.996427 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:09:29.998479 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:29.999606 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.001979 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.002591 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.003576 (+ 985us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:30.007097 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:09:30.007103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:09:30.007108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:09:30.007152 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:09:30.010763 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:09:30.011688 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:09:30.011692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:09:30.011696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:09:30.011843 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:09:30.012358 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:09:30.012362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:09:30.012366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:09:30.013181 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:09:30.017123 (+ 3942us) server_negotiation.cc:1035] Waiting for connection context
0929 01:09:30.017275 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 01:09:30.017497 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3384,"thread_start_us":3270,"threads_started":1}
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
I20260929 01:09:30.019328 17596 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33849
I20260929 01:09:30.019634 17596 heartbeater.cc:464] Registering TS with master...
I20260929 01:09:30.020434 17596 heartbeater.cc:511] Master 127.5.198.254:33849 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
I20260929 01:09:30.022106 17240 ts_manager.cc:194] Registered new tserver with Master: d703b9a5e94c4ec5b153ce6874ea5448 (127.5.198.194:39321)
I20260929 01:09:30.023195 17240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:53629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:09:30.056313 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.198.212:35083/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33849
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:09:30.173758 17453 heartbeater.cc:503] Master 127.5.198.254:33849 was elected leader, sending a full tablet report...
W20260929 01:09:30.402386 17602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:09:30.402706 17602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:09:30.402869 17602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:09:30.410058 17602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:09:30.410214 17602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:09:30.410399 17602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:09:30.420982 17602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.198.212:35083/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:09:30.422290 17602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:09:30.423568 17602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:09:30.437767 17607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:09:30.437717 17608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:09:30.438627 17610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:09:30.439296 17602 server_base.cc:1034] running on GCE node
I20260929 01:09:30.440075 17602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:09:30.441191 17602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:09:30.442423 17602 hybrid_clock.cc:648] HybridClock initialized: now 1790644170442361 us; error 55 us; skew 500 ppm
I20260929 01:09:30.442813 17602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:09:30 dist-test-slave-41cl krb5kdc[16764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644170, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:09:30.446084 17602 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:09:30.447705 17602 webserver.cc:460] Webserver started at http://127.5.198.195:43125/ using document root <none> and password file <none>
I20260929 01:09:30.448405 17602 fs_manager.cc:359] Metadata directory not provided
I20260929 01:09:30.448575 17602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:09:30.448892 17602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:09:30 dist-test-slave-41cl krb5kdc[16764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644170, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
I20260929 01:09:30.467455 17602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "77149af9e1a343fba780e833d13ccf9b"
format_stamp: "Formatted at 2026-09-29 01:09:30 on dist-test-slave-41cl"
server_key: "c59d337a317a24a85b65f2dd0e5c6fb7"
server_key_iv: "f842532768b53289419c52076bdb9fc6"
server_key_version: "kuduclusterkey@0"
I20260929 01:09:30.468613 17602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "77149af9e1a343fba780e833d13ccf9b"
format_stamp: "Formatted at 2026-09-29 01:09:30 on dist-test-slave-41cl"
server_key: "c59d337a317a24a85b65f2dd0e5c6fb7"
server_key_iv: "f842532768b53289419c52076bdb9fc6"
server_key_version: "kuduclusterkey@0"
I20260929 01:09:30.475515 17602 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260929 01:09:30.494328 17617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:09:30.495944 17602 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:09:30.496153 17602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "77149af9e1a343fba780e833d13ccf9b"
format_stamp: "Formatted at 2026-09-29 01:09:30 on dist-test-slave-41cl"
server_key: "c59d337a317a24a85b65f2dd0e5c6fb7"
server_key_iv: "f842532768b53289419c52076bdb9fc6"
server_key_version: "kuduclusterkey@0"
I20260929 01:09:30.496390 17602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:09:30.521901 17602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:09:30.526801 17602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:09:30.527125 17602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:09:30.528663 17602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:09:30.530290 17602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:09:30.530375 17602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:09:30.530534 17602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:09:30.530615 17602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:09:30.569005 17602 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:45597
I20260929 01:09:30.569039 17729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:45597 every 8 connection(s)
I20260929 01:09:30.570349 17602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:09:30.574631 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 17602
I20260929 01:09:30.574780 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:09:30.575213 5915 external_mini_cluster.cc:1428] Setting key efb719501b500e82714fd8f72476459d
Sep 29 01:09:30 dist-test-slave-41cl krb5kdc[16764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644170, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:09:30.592082 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:30.573055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:56373 (local address 127.5.198.254:33849)
0929 01:09:30.573532 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 01:09:30.573537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:09:30.575858 (+ 2321us) server_negotiation.cc:407] Connection header received
0929 01:09:30.577826 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:09:30.577854 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:09:30.577933 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:09:30.578034 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:09:30.579462 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.580208 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.582352 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.582786 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.582887 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:30.586251 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:09:30.586256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:09:30.586259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:09:30.586289 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:09:30.588862 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:09:30.589648 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:09:30.589651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:09:30.589653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:09:30.589778 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:09:30.590242 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:09:30.590245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:09:30.590248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:09:30.590539 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:09:30.590680 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 01:09:30.591609 (+ 929us) server_negotiation.cc:299] Negotiation successful
0929 01:09:30.591782 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":250,"threads_started":1}
I20260929 01:09:30.593007 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:30.573644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33849 (local address 127.5.198.195:56373)
0929 01:09:30.575691 (+ 2047us) negotiation.cc:107] Waiting for socket to connect
0929 01:09:30.575733 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:09:30.577534 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:09:30.578332 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:09:30.578345 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:09:30.578757 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:09:30.579281 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:09:30.579296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.580392 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.580396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:09:30.582239 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:09:30.582251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.583026 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.583031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:09:30.583220 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:30.583962 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:09:30.583988 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:09:30.586027 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:09:30.589000 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:09:30.589007 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:09:30.589018 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:09:30.589529 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:09:30.589936 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:09:30.589939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:09:30.589941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:09:30.590123 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:09:30.590699 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:09:30.590708 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:09:30.591198 (+ 490us) client_negotiation.cc:769] Sending connection context
0929 01:09:30.591457 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 01:09:30.591824 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":269,"threads_started":1}
I20260929 01:09:30.594559 17730 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33849
I20260929 01:09:30.594841 17730 heartbeater.cc:464] Registering TS with master...
I20260929 01:09:30.595499 17730 heartbeater.cc:511] Master 127.5.198.254:33849 requested a full tablet report, sending...
I20260929 01:09:30.597043 17240 ts_manager.cc:194] Registered new tserver with Master: 77149af9e1a343fba780e833d13ccf9b (127.5.198.195:45597)
I20260929 01:09:30.597949 17240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:56373
I20260929 01:09:30.603586 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:09:30.621212 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:30.607914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39764 (local address 127.5.198.254:33849)
0929 01:09:30.608103 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 01:09:30.608112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:09:30.608515 (+ 403us) server_negotiation.cc:407] Connection header received
0929 01:09:30.608728 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:09:30.608733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:09:30.608812 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:09:30.608912 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:09:30.610726 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.611444 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.612884 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.613304 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.613440 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:30.615943 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:09:30.615948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:09:30.615951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:09:30.615991 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:09:30.618617 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:09:30.619260 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:09:30.619268 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:09:30.619274 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:09:30.619391 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:09:30.619767 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:09:30.619772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:09:30.619776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:09:30.620029 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:09:30.620277 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 01:09:30.620651 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 01:09:30.620889 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 01:09:30.625361 17240 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:09:30.633339 17240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:09:30.670506 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:30.646557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37318 (local address 127.5.198.194:39321)
0929 01:09:30.661344 (+ 14787us) server_negotiation.cc:206] Beginning negotiation
0929 01:09:30.661352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:09:30.661382 (+ 30us) server_negotiation.cc:407] Connection header received
0929 01:09:30.661495 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:09:30.661502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:09:30.661806 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:09:30.662904 (+ 1098us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:09:30.663835 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.665484 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.667889 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.669333 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.669484 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:30.669689 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 01:09:30.669981 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 01:09:30.670138 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14616,"thread_start_us":14487,"threads_started":1}
I20260929 01:09:30.670532 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:30.646348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:39321 (local address 127.0.0.1:37318)
0929 01:09:30.660523 (+ 14175us) negotiation.cc:107] Waiting for socket to connect
0929 01:09:30.660568 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 01:09:30.660744 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:09:30.662113 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:09:30.662123 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:09:30.662158 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:09:30.663670 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:09:30.663683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.665623 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.665628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:09:30.667731 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:09:30.667742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.669540 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.669547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:09:30.669722 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:30.669778 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 01:09:30.669920 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 01:09:30.670104 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13897,"mutex_wait_us":48,"thread_start_us":15194,"threads_started":1}
I20260929 01:09:30.671942 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:30.661374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.5.198.193:45157)
0929 01:09:30.662127 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0929 01:09:30.662134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:09:30.663392 (+ 1258us) server_negotiation.cc:407] Connection header received
0929 01:09:30.663740 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:09:30.663747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:09:30.663971 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:09:30.664986 (+ 1015us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:09:30.665229 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.666715 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.668839 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.670565 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.670714 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:30.670932 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 01:09:30.671456 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 01:09:30.671609 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":433,"threads_started":1}
I20260929 01:09:30.671985 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:30.647376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35788 (local address 127.5.198.195:45597)
0929 01:09:30.659670 (+ 12294us) server_negotiation.cc:206] Beginning negotiation
0929 01:09:30.659684 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:09:30.661507 (+ 1823us) server_negotiation.cc:407] Connection header received
0929 01:09:30.661746 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:09:30.661757 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:09:30.661990 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:09:30.662163 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:09:30.663928 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.665480 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.668123 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.669512 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.669618 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:30.669800 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 01:09:30.671454 (+ 1654us) server_negotiation.cc:299] Negotiation successful
0929 01:09:30.671599 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12027,"thread_start_us":12306,"threads_started":1}
I20260929 01:09:30.672627 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:30.646729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:45597 (local address 127.0.0.1:35788)
0929 01:09:30.661367 (+ 14638us) negotiation.cc:107] Waiting for socket to connect
0929 01:09:30.661391 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:09:30.661538 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:09:30.662257 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:09:30.662263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:09:30.662292 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:09:30.663750 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:09:30.663763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.665691 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.665696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:09:30.667983 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:09:30.667996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.669941 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.669947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:09:30.671157 (+ 1210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:30.671278 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 01:09:30.672152 (+ 874us) client_negotiation.cc:240] Negotiation successful
0929 01:09:30.672273 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14520,"mutex_wait_us":171,"thread_start_us":13360,"threads_started":1}
I20260929 01:09:30.672662 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:30.662153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:45157 (local address 127.0.0.1:46530)
0929 01:09:30.663254 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0929 01:09:30.663267 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:09:30.663544 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:09:30.664134 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:09:30.664139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:09:30.664163 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:09:30.664591 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:09:30.664602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.666863 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.666868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:09:30.668693 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:09:30.668703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.671071 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.671078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:09:30.671277 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:30.671299 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:09:30.671397 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 01:09:30.671512 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"mutex_wait_us":46,"thread_start_us":825,"threads_started":1}
I20260929 01:09:30.674698 17531 tablet_service.cc:1467] Processing CreateTablet for tablet 9015ee3d31a44389b7846480f0c91156 (DEFAULT_TABLE table=test-table [id=6a40bc70bbe24ff8b0950e76ba52dc2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:09:30.675951 17531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9015ee3d31a44389b7846480f0c91156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:09:30.676204 17388 tablet_service.cc:1467] Processing CreateTablet for tablet 9015ee3d31a44389b7846480f0c91156 (DEFAULT_TABLE table=test-table [id=6a40bc70bbe24ff8b0950e76ba52dc2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:09:30.676204 17665 tablet_service.cc:1467] Processing CreateTablet for tablet 9015ee3d31a44389b7846480f0c91156 (DEFAULT_TABLE table=test-table [id=6a40bc70bbe24ff8b0950e76ba52dc2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:09:30.677668 17388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9015ee3d31a44389b7846480f0c91156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:09:30.677662 17665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9015ee3d31a44389b7846480f0c91156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:09:30.689067 17749 tablet_bootstrap.cc:492] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448: Bootstrap starting.
I20260929 01:09:30.691821 17749 tablet_bootstrap.cc:654] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448: Neither blocks nor log segments found. Creating new log.
I20260929 01:09:30.692684 17750 tablet_bootstrap.cc:492] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b: Bootstrap starting.
I20260929 01:09:30.693177 17749 log.cc:824] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448: Log is configured to *not* fsync() on all Append() calls
I20260929 01:09:30.694540 17751 tablet_bootstrap.cc:492] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1: Bootstrap starting.
I20260929 01:09:30.696115 17749 tablet_bootstrap.cc:492] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448: No bootstrap required, opened a new log
I20260929 01:09:30.696399 17749 ts_tablet_manager.cc:1397] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260929 01:09:30.696681 17750 tablet_bootstrap.cc:654] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b: Neither blocks nor log segments found. Creating new log.
I20260929 01:09:30.697888 17750 log.cc:824] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:09:30.699479 17751 tablet_bootstrap.cc:654] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1: Neither blocks nor log segments found. Creating new log.
I20260929 01:09:30.700686 17751 log.cc:824] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1: Log is configured to *not* fsync() on all Append() calls
I20260929 01:09:30.700979 17750 tablet_bootstrap.cc:492] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b: No bootstrap required, opened a new log
I20260929 01:09:30.701261 17750 ts_tablet_manager.cc:1397] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260929 01:09:30.703994 17751 tablet_bootstrap.cc:492] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1: No bootstrap required, opened a new log
I20260929 01:09:30.704234 17751 ts_tablet_manager.cc:1397] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260929 01:09:30.704957 17749 raft_consensus.cc:348] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d703b9a5e94c4ec5b153ce6874ea5448" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39321 } } peers { permanent_uuid: "77149af9e1a343fba780e833d13ccf9b" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45597 } } peers { permanent_uuid: "a1c11acaf3bb4e4daa5de70062b3dbc1" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45157 } }
I20260929 01:09:30.705361 17749 raft_consensus.cc:374] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:09:30.705471 17749 raft_consensus.cc:729] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d703b9a5e94c4ec5b153ce6874ea5448, State: Initialized, Role: FOLLOWER
I20260929 01:09:30.706082 17749 consensus_queue.cc:260] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d703b9a5e94c4ec5b153ce6874ea5448" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39321 } } peers { permanent_uuid: "77149af9e1a343fba780e833d13ccf9b" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45597 } } peers { permanent_uuid: "a1c11acaf3bb4e4daa5de70062b3dbc1" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45157 } }
I20260929 01:09:30.707175 17596 heartbeater.cc:503] Master 127.5.198.254:33849 was elected leader, sending a full tablet report...
I20260929 01:09:30.708881 17749 ts_tablet_manager.cc:1428] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260929 01:09:30.711793 17750 raft_consensus.cc:348] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d703b9a5e94c4ec5b153ce6874ea5448" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39321 } } peers { permanent_uuid: "77149af9e1a343fba780e833d13ccf9b" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45597 } } peers { permanent_uuid: "a1c11acaf3bb4e4daa5de70062b3dbc1" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45157 } }
I20260929 01:09:30.712188 17750 raft_consensus.cc:374] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:09:30.712287 17750 raft_consensus.cc:729] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77149af9e1a343fba780e833d13ccf9b, State: Initialized, Role: FOLLOWER
I20260929 01:09:30.712877 17750 consensus_queue.cc:260] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d703b9a5e94c4ec5b153ce6874ea5448" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39321 } } peers { permanent_uuid: "77149af9e1a343fba780e833d13ccf9b" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45597 } } peers { permanent_uuid: "a1c11acaf3bb4e4daa5de70062b3dbc1" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45157 } }
I20260929 01:09:30.713923 17730 heartbeater.cc:503] Master 127.5.198.254:33849 was elected leader, sending a full tablet report...
I20260929 01:09:30.714570 17750 ts_tablet_manager.cc:1428] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 01:09:30.716704 17751 raft_consensus.cc:348] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d703b9a5e94c4ec5b153ce6874ea5448" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39321 } } peers { permanent_uuid: "77149af9e1a343fba780e833d13ccf9b" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45597 } } peers { permanent_uuid: "a1c11acaf3bb4e4daa5de70062b3dbc1" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45157 } }
I20260929 01:09:30.717064 17751 raft_consensus.cc:374] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:09:30.717171 17751 raft_consensus.cc:729] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c11acaf3bb4e4daa5de70062b3dbc1, State: Initialized, Role: FOLLOWER
I20260929 01:09:30.717813 17751 consensus_queue.cc:260] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d703b9a5e94c4ec5b153ce6874ea5448" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39321 } } peers { permanent_uuid: "77149af9e1a343fba780e833d13ccf9b" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45597 } } peers { permanent_uuid: "a1c11acaf3bb4e4daa5de70062b3dbc1" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45157 } }
I20260929 01:09:30.719125 17751 ts_tablet_manager.cc:1428] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
W20260929 01:09:30.743216 17597 tablet.cc:2369] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:09:30.822103 17731 tablet.cc:2369] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:09:30.886615 17454 tablet.cc:2369] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:09:30.939030 17755 raft_consensus.cc:482] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:09:30.939474 17755 raft_consensus.cc:504] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d703b9a5e94c4ec5b153ce6874ea5448" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39321 } } peers { permanent_uuid: "77149af9e1a343fba780e833d13ccf9b" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45597 } } peers { permanent_uuid: "a1c11acaf3bb4e4daa5de70062b3dbc1" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45157 } }
I20260929 01:09:30.941476 17755 leader_election.cc:288] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77149af9e1a343fba780e833d13ccf9b (127.5.198.195:45597), a1c11acaf3bb4e4daa5de70062b3dbc1 (127.5.198.193:45157)
I20260929 01:09:30.949194 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:30.941980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:56589 (local address 127.5.198.195:45597)
0929 01:09:30.942171 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 01:09:30.942176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:09:30.943180 (+ 1004us) server_negotiation.cc:407] Connection header received
0929 01:09:30.943427 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:09:30.943433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:09:30.943521 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:09:30.943587 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:09:30.944419 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.945443 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.947311 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.948183 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.948306 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:30.948356 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 01:09:30.948766 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 01:09:30.948940 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:09:30.949306 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:30.942385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:45597 (local address 127.5.198.194:56589)
0929 01:09:30.943032 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0929 01:09:30.943053 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:09:30.943216 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:09:30.943805 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:09:30.943809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:09:30.943840 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:09:30.944231 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:09:30.944245 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.945576 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.945582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:09:30.947160 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:09:30.947171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.948397 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.948405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:09:30.948592 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:30.948628 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 01:09:30.948724 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 01:09:30.948865 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":2,"thread_start_us":299,"threads_started":1}
I20260929 01:09:30.949508 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:30.943347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:36611 (local address 127.5.198.193:45157)
0929 01:09:30.943570 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 01:09:30.943577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:09:30.943797 (+ 220us) server_negotiation.cc:407] Connection header received
0929 01:09:30.943922 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:09:30.943929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:09:30.944011 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:09:30.944110 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:09:30.944905 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.945748 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.947600 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.948515 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.948668 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:30.948748 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 01:09:30.949107 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 01:09:30.949238 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 01:09:30.949854 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:30.943068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:45157 (local address 127.5.198.194:36611)
0929 01:09:30.943683 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0929 01:09:30.943698 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:09:30.943811 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:09:30.944171 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:09:30.944174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:09:30.944201 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:09:30.944657 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:09:30.944670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.945906 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.945909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:09:30.947424 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:09:30.947437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:09:30.948744 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:09:30.948750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:09:30.948930 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:30.948960 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 01:09:30.949059 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 01:09:30.949171 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":339,"threads_started":1}
I20260929 01:09:30.950367 17685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9015ee3d31a44389b7846480f0c91156" candidate_uuid: "d703b9a5e94c4ec5b153ce6874ea5448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77149af9e1a343fba780e833d13ccf9b" is_pre_election: true
I20260929 01:09:30.950783 17685 raft_consensus.cc:2454] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d703b9a5e94c4ec5b153ce6874ea5448 in term 0.
I20260929 01:09:30.951403 17479 leader_election.cc:302] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77149af9e1a343fba780e833d13ccf9b, d703b9a5e94c4ec5b153ce6874ea5448; no voters:
I20260929 01:09:30.951473 17408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9015ee3d31a44389b7846480f0c91156" candidate_uuid: "d703b9a5e94c4ec5b153ce6874ea5448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1c11acaf3bb4e4daa5de70062b3dbc1" is_pre_election: true
I20260929 01:09:30.951783 17755 raft_consensus.cc:2781] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:09:30.951972 17755 raft_consensus.cc:482] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:09:30.952013 17408 raft_consensus.cc:2454] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d703b9a5e94c4ec5b153ce6874ea5448 in term 0.
I20260929 01:09:30.952070 17755 raft_consensus.cc:3037] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:09:30.954712 17755 raft_consensus.cc:504] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d703b9a5e94c4ec5b153ce6874ea5448" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39321 } } peers { permanent_uuid: "77149af9e1a343fba780e833d13ccf9b" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45597 } } peers { permanent_uuid: "a1c11acaf3bb4e4daa5de70062b3dbc1" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45157 } }
I20260929 01:09:30.955314 17755 leader_election.cc:288] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [CANDIDATE]: Term 1 election: Requested vote from peers 77149af9e1a343fba780e833d13ccf9b (127.5.198.195:45597), a1c11acaf3bb4e4daa5de70062b3dbc1 (127.5.198.193:45157)
I20260929 01:09:30.955753 17685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9015ee3d31a44389b7846480f0c91156" candidate_uuid: "d703b9a5e94c4ec5b153ce6874ea5448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77149af9e1a343fba780e833d13ccf9b"
I20260929 01:09:30.955869 17408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9015ee3d31a44389b7846480f0c91156" candidate_uuid: "d703b9a5e94c4ec5b153ce6874ea5448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1c11acaf3bb4e4daa5de70062b3dbc1"
I20260929 01:09:30.955938 17685 raft_consensus.cc:3037] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:09:30.956090 17408 raft_consensus.cc:3037] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:09:30.958114 17685 raft_consensus.cc:2454] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d703b9a5e94c4ec5b153ce6874ea5448 in term 1.
I20260929 01:09:30.958408 17408 raft_consensus.cc:2454] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d703b9a5e94c4ec5b153ce6874ea5448 in term 1.
I20260929 01:09:30.958508 17479 leader_election.cc:302] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77149af9e1a343fba780e833d13ccf9b, d703b9a5e94c4ec5b153ce6874ea5448; no voters:
I20260929 01:09:30.958829 17755 raft_consensus.cc:2781] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:09:30.959360 17755 raft_consensus.cc:686] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [term 1 LEADER]: Becoming Leader. State: Replica: d703b9a5e94c4ec5b153ce6874ea5448, State: Running, Role: LEADER
I20260929 01:09:30.959801 17755 consensus_queue.cc:237] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d703b9a5e94c4ec5b153ce6874ea5448" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39321 } } peers { permanent_uuid: "77149af9e1a343fba780e833d13ccf9b" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45597 } } peers { permanent_uuid: "a1c11acaf3bb4e4daa5de70062b3dbc1" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45157 } }
I20260929 01:09:30.964741 17239 catalog_manager.cc:5495] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 reported cstate change: term changed from 0 to 1, leader changed from <none> to d703b9a5e94c4ec5b153ce6874ea5448 (127.5.198.194). New cstate: current_term: 1 leader_uuid: "d703b9a5e94c4ec5b153ce6874ea5448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d703b9a5e94c4ec5b153ce6874ea5448" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77149af9e1a343fba780e833d13ccf9b" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1c11acaf3bb4e4daa5de70062b3dbc1" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45157 } health_report { overall_health: UNKNOWN } } }
I20260929 01:09:31.025712 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:31.020419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.5.198.194:39321)
0929 01:09:31.020631 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 01:09:31.020635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:09:31.020654 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:09:31.020895 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:09:31.020900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:09:31.020991 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:09:31.021116 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:09:31.021470 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:31.022210 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:31.023677 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:31.024133 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:31.024238 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:31.024735 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:09:31.025263 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:09:31.025378 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 01:09:31.025446 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 01:09:31.025508 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 01:09:31.033828 17685 raft_consensus.cc:1261] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b [term 1 FOLLOWER]: Refusing update from remote peer d703b9a5e94c4ec5b153ce6874ea5448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:09:31.033989 17408 raft_consensus.cc:1261] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1 [term 1 FOLLOWER]: Refusing update from remote peer d703b9a5e94c4ec5b153ce6874ea5448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:09:31.034834 17760 consensus_queue.cc:1035] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77149af9e1a343fba780e833d13ccf9b" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 01:09:31.035720 17755 consensus_queue.cc:1035] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1c11acaf3bb4e4daa5de70062b3dbc1" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 45157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:09:31.048628 17764 mvcc.cc:205] Tried to move back new op lower bound from 7334478524543533056 to 7334478524255158272. Current Snapshot: MvccSnapshot[applied={T|T < 7334478524543533056}]
I20260929 01:09:31.060134 17240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:09:31.060410 17240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:09:31.063889 17240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0357ba9502134d17847b40d44a05841e: Sending DeleteTablet for 3 replicas of tablet 9015ee3d31a44389b7846480f0c91156
I20260929 01:09:31.065161 17665 tablet_service.cc:1514] Processing DeleteTablet for tablet 9015ee3d31a44389b7846480f0c91156 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:09:31 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:35788
I20260929 01:09:31.065207 17388 tablet_service.cc:1514] Processing DeleteTablet for tablet 9015ee3d31a44389b7846480f0c91156 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:09:31 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:46530
I20260929 01:09:31.065222 17531 tablet_service.cc:1514] Processing DeleteTablet for tablet 9015ee3d31a44389b7846480f0c91156 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:09:31 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:37318
I20260929 01:09:31.067868 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 17320
I20260929 01:09:31.077085 17774 tablet_replica.cc:330] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1: stopping tablet replica
I20260929 01:09:31.077574 17774 raft_consensus.cc:2229] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:09:31.078136 17774 raft_consensus.cc:2258] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:09:31.078927 17775 tablet_replica.cc:330] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448: stopping tablet replica
I20260929 01:09:31.079483 17775 raft_consensus.cc:2229] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:09:31.080181 17775 raft_consensus.cc:2258] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:09:31.081070 17773 tablet_replica.cc:330] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b: stopping tablet replica
I20260929 01:09:31.083062 17775 ts_tablet_manager.cc:1905] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:09:31.085764 17774 ts_tablet_manager.cc:1905] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:09:31.090437 17775 ts_tablet_manager.cc:1918] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:09:31.091014 17773 raft_consensus.cc:2229] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:09:31.091590 17773 raft_consensus.cc:2258] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:09:31.092025 17775 log.cc:1196] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/9015ee3d31a44389b7846480f0c91156
I20260929 01:09:31.092653 17775 ts_tablet_manager.cc:1939] T 9015ee3d31a44389b7846480f0c91156 P d703b9a5e94c4ec5b153ce6874ea5448: Deleting consensus metadata
Sep 29 01:09:31 dist-test-slave-41cl krb5kdc[16764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:09:31.093716 17773 ts_tablet_manager.cc:1905] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:09:31.095381 17227 catalog_manager.cc:4841] TS d703b9a5e94c4ec5b153ce6874ea5448 (127.5.198.194:39321): tablet 9015ee3d31a44389b7846480f0c91156 (table test-table [id=6a40bc70bbe24ff8b0950e76ba52dc2b]) successfully deleted
I20260929 01:09:31.097729 17774 ts_tablet_manager.cc:1918] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:09:31.097971 17774 log.cc:1196] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/9015ee3d31a44389b7846480f0c91156
I20260929 01:09:31.098589 17774 ts_tablet_manager.cc:1939] T 9015ee3d31a44389b7846480f0c91156 P a1c11acaf3bb4e4daa5de70062b3dbc1: Deleting consensus metadata
I20260929 01:09:31.100739 17225 catalog_manager.cc:4841] TS a1c11acaf3bb4e4daa5de70062b3dbc1 (127.5.198.193:45157): tablet 9015ee3d31a44389b7846480f0c91156 (table test-table [id=6a40bc70bbe24ff8b0950e76ba52dc2b]) successfully deleted
I20260929 01:09:31.100833 17773 ts_tablet_manager.cc:1918] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:09:31.100982 17773 log.cc:1196] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/9015ee3d31a44389b7846480f0c91156
I20260929 01:09:31.101445 17773 ts_tablet_manager.cc:1939] T 9015ee3d31a44389b7846480f0c91156 P 77149af9e1a343fba780e833d13ccf9b: Deleting consensus metadata
I20260929 01:09:31.103051 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:31.069666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46534 (local address 127.5.198.193:45157)
0929 01:09:31.069902 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 01:09:31.069911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:09:31.083495 (+ 13584us) server_negotiation.cc:407] Connection header received
0929 01:09:31.083712 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:09:31.083718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:09:31.083802 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:09:31.083903 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:09:31.085760 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:31.086797 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:31.088736 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:31.089402 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:31.089520 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:31.094934 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:09:31.094952 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:09:31.094971 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:09:31.095013 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:09:31.099132 (+ 4119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:09:31.100100 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:09:31.100109 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:09:31.100113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:09:31.100266 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:09:31.101308 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:09:31.101313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:09:31.101317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:09:31.101777 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:09:31.102070 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 01:09:31.102414 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 01:09:31.102660 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 01:09:31.103071 17225 catalog_manager.cc:4841] TS 77149af9e1a343fba780e833d13ccf9b (127.5.198.195:45597): tablet 9015ee3d31a44389b7846480f0c91156 (table test-table [id=6a40bc70bbe24ff8b0950e76ba52dc2b]) successfully deleted
I20260929 01:09:31.103787 17448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:09:31.230062 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 17320
I20260929 01:09:31.247359 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 17459
Sep 29 01:09:31 dist-test-slave-41cl krb5kdc[16764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:09:31.263245 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:31.248166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.5.198.194:39321)
0929 01:09:31.248332 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 01:09:31.248339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:09:31.248363 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:09:31.248627 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:09:31.248632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:09:31.248710 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:09:31.248818 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:09:31.250075 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:31.250888 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:31.252213 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:31.252709 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:31.252868 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:31.257001 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:09:31.257016 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:09:31.257035 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:09:31.257074 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:09:31.260170 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:09:31.260938 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:09:31.260945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:09:31.260951 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:09:31.261106 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:09:31.261539 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:09:31.261543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:09:31.261545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:09:31.261889 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:09:31.262243 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 01:09:31.262671 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 01:09:31.262894 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 01:09:31.263950 17591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:09:31.376912 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 17459
I20260929 01:09:31.395290 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 17602
Sep 29 01:09:31 dist-test-slave-41cl krb5kdc[16764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:09:31.411016 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:09:31.396104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.5.198.195:45597)
0929 01:09:31.396271 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 01:09:31.396275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:09:31.396304 (+ 29us) server_negotiation.cc:407] Connection header received
0929 01:09:31.396579 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:09:31.396583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:09:31.396646 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:09:31.396727 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:09:31.398134 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:31.398856 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:31.400411 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:09:31.400930 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:09:31.401035 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:09:31.405596 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:09:31.405609 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:09:31.405625 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:09:31.405665 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:09:31.408400 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:09:31.409121 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:09:31.409130 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:09:31.409132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:09:31.409267 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:09:31.409623 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:09:31.409626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:09:31.409629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:09:31.409991 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:09:31.410190 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 01:09:31.410512 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 01:09:31.410661 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:09:31.411695 17725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:09:31.499460 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 17602
I20260929 01:09:31.516291 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 17208
I20260929 01:09:31.517596 17270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:09:31.631479 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 17208
I20260929 01:09:31.656235 5915 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 01:09:31.991292 5915 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 01:09:31.991485 5915 mini_ranger.cc:66] Stopping Ranger...
I20260929 01:09:32.336794 5915 mini_ranger.cc:68] Stopped Ranger
2026-09-29 01:09:32.337 UTC [16794] LOG: received smart shutdown request
2026-09-29 01:09:32.340 UTC [16794] LOG: background worker "logical replication launcher" (PID 16803) exited with exit code 1
2026-09-29 01:09:32.340 UTC [16798] LOG: shutting down
2026-09-29 01:09:32.538 UTC [16794] LOG: database system is shut down
2026-09-29T01:09:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49543 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:09:32 dist-test-slave-41cl krb5kdc[17789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:09:32 dist-test-slave-41cl krb5kdc[17789](info): set up 2 sockets
Sep 29 01:09:32 dist-test-slave-41cl krb5kdc[17789](info): commencing operation
krb5kdc: starting...
W20260929 01:09:33.764859 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:09:33 dist-test-slave-41cl krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:09:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:09:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.198.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.198.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.198.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/rangeradmin_127.5.198.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.198.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/rangeradmin_127.5.198.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.198.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.198.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.198.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/rangerlookup_127.5.198.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.198.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/rangerlookup_127.5.198.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.198.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.198.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.198.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/HTTP_127.5.198.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.198.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/HTTP_127.5.198.212@KRBTEST.COM.keytab.
I20260929 01:09:34.103453 5915 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskd3rvDY/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/postgres -l logfile start
2026-09-29 01:09:36.720 UTC [17819] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 01:09:36.720 UTC [17819] LOG: listening on IPv4 address "127.5.198.212", port 43885
2026-09-29 01:09:36.724 UTC [17819] LOG: listening on Unix socket "/tmp/.s.PGSQL.43885"
2026-09-29 01:09:36.745 UTC [17822] LOG: database system was shut down at 2026-09-29 01:09:35 UTC
2026-09-29 01:09:36.749 UTC [17819] LOG: database system is ready to accept connections
I20260929 01:09:37.708644 5915 mini_postgres.cc:96] Postgres bound to 43885
2026-09-29 01:09:37.731 UTC [17830] FATAL: database "slave" does not exist
127.5.198.212:43885 - accepting connections
I20260929 01:09:37.731768 5915 mini_ranger.cc:161] Starting Ranger...
I20260929 01:09:37.756968 5915 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 01:09:38.304796 5915 mini_ranger.cc:87] Created ranger Postgres database
I20260929 01:09:38.304940 5915 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-admin
2026-09-29 01:09:38,584 [I] DB FLAVOR :POSTGRES
2026-09-29 01:09:38,584 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 01:09:38,584 [I] Checking connection..
2026-09-29 01:09:38,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 01:09:38,831 [I] Checking connection passed.
2026-09-29 01:09:38,832 [I] --------- Verifying version history table ---------
2026-09-29 01:09:38,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:09:39,091 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 01:09:39,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:09:39,317 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 01:09:39,317 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 01:09:39,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 01:09:39.570 UTC [17894] WARNING: there is no transaction in progress
2026-09-29 01:09:39,576 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 01:09:39,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:09:39,835 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 01:09:39,835 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 01:09:39,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 01:09:40,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 01:09:40,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-41cl.c.gcp-upstream.internal','N') ;"
2026-09-29 01:09:40,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 01:09:40,822 [I] Table x_portal_user does not exist in database ranger
2026-09-29 01:09:40,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 01:09:41,055 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 01:09:41,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 01:09:41,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 01:09:41,556 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 01:09:41,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 01:09:42.581 UTC [18002] WARNING: there is no transaction in progress
2026-09-29 01:09:42.599 UTC [18002] WARNING: there is no transaction in progress
2026-09-29 01:09:42.619 UTC [18002] WARNING: there is no transaction in progress
2026-09-29 01:09:42.638 UTC [18002] WARNING: there is no transaction in progress
2026-09-29 01:09:42.654 UTC [18002] WARNING: there is no transaction in progress
2026-09-29 01:09:42.674 UTC [18002] WARNING: there is no transaction in progress
2026-09-29 01:09:42.738 UTC [18002] WARNING: there is no transaction in progress
2026-09-29 01:09:42.746 UTC [18002] WARNING: there is no transaction in progress
2026-09-29 01:09:42.763 UTC [18002] WARNING: there is no transaction in progress
2026-09-29 01:09:42.773 UTC [18002] WARNING: there is no transaction in progress
2026-09-29 01:09:42.784 UTC [18002] WARNING: there is no transaction in progress
2026-09-29 01:09:42.799 UTC [18002] WARNING: there is no transaction in progress
2026-09-29 01:09:42.808 UTC [18002] WARNING: there is no transaction in progress
2026-09-29 01:09:42.815 UTC [18002] WARNING: there is no transaction in progress
2026-09-29 01:09:42.822 UTC [18002] WARNING: there is no transaction in progress
2026-09-29 01:09:42.829 UTC [18002] WARNING: there is no transaction in progress
2026-09-29 01:09:43,550 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 01:09:43,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 01:09:43,784 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 01:09:43,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 01:09:44,039 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 01:09:44,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 01:09:44,324 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 01:09:44,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 01:09:44,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 01:09:44,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-41cl.c.gcp-upstream.internal';"
2026-09-29 01:09:45,073 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 01:09:45,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 01:09:45,325 [I] DB_PATCHES have already been applied
I20260929 01:09:45.331533 5915 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-admin:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskd3rvDY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 01:09:45.331622 5915 mini_ranger.cc:193] Using host: 127.5.198.212
I20260929 01:09:45.346057 5915 mini_ranger.cc:227] Ranger admin URL: http://127.5.198.212:40791
Sep 29, 2026 1:09:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 1:09:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 1:09:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 1:09:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 1:09:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 1:09:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 1:09:46 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 01:09:46.475785 5915 mini_ranger.cc:160] Time spent starting Ranger: real 8.744s user 0.000s sys 0.037s
Sep 29, 2026 1:10:05 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 1:10:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 1:10:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 1:10:05 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 1:10:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 1:10:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 1:10:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 1:10:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 1:10:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 1:10:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 1:10:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 1:10:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 1:10:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 1:10:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 1:10:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 1:10:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 1:10:05 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 1:10:06 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 01:10:07.282346 5915 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.198.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.198.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.198.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/rangerkms_127.5.198.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.198.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/rangerkms_127.5.198.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.198.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.198.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.198.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/HTTP_127.5.198.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.198.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/HTTP_127.5.198.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 01:10:07 dist-test-slave-41cl krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644207, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 01:10:07.468779 5915 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 01:10:07.489818 5915 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 01:10:08.548238 5915 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 01:10:08.548362 5915 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-kms
I20260929 01:10:08.548417 5915 mini_ranger_kms.cc:227] Using postgres at 127.5.198.212:43885
2026-09-29 01:10:08,908 [I] DB FLAVOR :POSTGRES
2026-09-29 01:10:08,908 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 01:10:08,908 [I] Checking connection
2026-09-29 01:10:08,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 01:10:09,181 [I] connection success
2026-09-29 01:10:09,182 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 01:10:09,182 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 01:10:09,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 01:10:09,412 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 01:10:09,412 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 01:10:09,412 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 01:10:09,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.198.212:43885/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 01:10:09,668 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 01:10:10.137952 5915 mini_ranger_kms.cc:310] Created kms service
I20260929 01:10:10.349143 5915 mini_ranger_kms.cc:326] Created kudu user
I20260929 01:10:10.392856 5915 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 01:10:10 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644207, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
I20260929 01:10:10.819067 5915 mini_ranger_kms.cc:384] Added ranger policy
I20260929 01:10:10.819377 5915 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-kms:/tmp/dist-test-taskd3rvDY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskd3rvDY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 01:10:10.819427 5915 mini_ranger_kms.cc:240] Using host: 127.5.198.212
I20260929 01:10:10.834434 5915 mini_ranger_kms.cc:276] Ranger KMS PID: 18182
I20260929 01:10:10.834563 5915 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.198.212:50541
Sep 29, 2026 1:10:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 1:10:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 1:10:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 1:10:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 1:10:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 1:10:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 01:10:13.118754 5915 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.650s user 0.080s sys 0.292s
I20260929 01:10:13.118996 5915 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 01:10:13.119167 5915 mini_ranger_kms.cc:401] 127.5.198.212:50541/kms/v1/keys
Sep 29 01:10:13 dist-test-slave-41cl krb5kdc[17789](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790644213, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.198.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:10:13 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644213, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.198.212@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
Sep 29 01:10:14 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644213, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.198.212@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
Sep 29 01:10:16 dist-test-slave-41cl krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:10:16.501009 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:44073
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:44073
--ranger_config_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.198.212:50541/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:10:16.839175 18233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:16.839462 18233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:16.839550 18233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:16.846935 18233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:10:16.847054 18233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:16.847100 18233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:16.847157 18233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:10:16.847205 18233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:10:16.858660 18233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.198.212:50541/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:44073
--ranger_config_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:44073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:16.860036 18233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:16.861426 18233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:16.873776 18241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:16.874046 18233 server_base.cc:1034] running on GCE node
W20260929 01:10:16.873863 18239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:16.874511 18238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:16.875352 18233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:16.876811 18233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:16.878031 18233 hybrid_clock.cc:648] HybridClock initialized: now 1790644216877978 us; error 49 us; skew 500 ppm
I20260929 01:10:16.878393 18233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:16 dist-test-slave-41cl krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644216, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:16.881481 18233 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:10:16.882800 18233 webserver.cc:460] Webserver started at http://127.5.198.254:38895/ using document root <none> and password file <none>
I20260929 01:10:16.883411 18233 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:16.883502 18233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:16.883751 18233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:10:16 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644216, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
I20260929 01:10:16.945461 18233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "ee1a8cc38804409e89fa5ac821673678"
format_stamp: "Formatted at 2026-09-29 01:10:16 on dist-test-slave-41cl"
server_key: "90c039075ca7fb91affd1ff25d2deba8"
server_key_iv: "9b8d085b7f5cef52328bcaab5d4bf733"
server_key_version: "kuduclusterkey@0"
I20260929 01:10:16.946514 18233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "ee1a8cc38804409e89fa5ac821673678"
format_stamp: "Formatted at 2026-09-29 01:10:16 on dist-test-slave-41cl"
server_key: "90c039075ca7fb91affd1ff25d2deba8"
server_key_iv: "9b8d085b7f5cef52328bcaab5d4bf733"
server_key_version: "kuduclusterkey@0"
I20260929 01:10:16.952246 18233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 01:10:16.972891 18248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:16.974421 18233 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:10:16.974581 18233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "ee1a8cc38804409e89fa5ac821673678"
format_stamp: "Formatted at 2026-09-29 01:10:16 on dist-test-slave-41cl"
server_key: "90c039075ca7fb91affd1ff25d2deba8"
server_key_iv: "9b8d085b7f5cef52328bcaab5d4bf733"
server_key_version: "kuduclusterkey@0"
I20260929 01:10:16.974793 18233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:17.002077 18233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:17.006187 18233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:17.006465 18233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:17.025276 18233 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:44073
I20260929 01:10:17.025314 18299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:44073 every 8 connection(s)
I20260929 01:10:17.026787 18233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:10:17.029739 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 18233
I20260929 01:10:17.029953 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:10:17.030578 5915 external_mini_cluster.cc:1428] Setting key baea132d768dd1bb85d735d87707c182
I20260929 01:10:17.030689 18300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:10:17.045279 18300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678: Bootstrap starting.
Sep 29 01:10:17 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:17.048412 18300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678: Neither blocks nor log segments found. Creating new log.
I20260929 01:10:17.049494 18300 log.cc:824] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:17.052268 18300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678: No bootstrap required, opened a new log
I20260929 01:10:17.054888 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:17.032541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35312 (local address 127.5.198.254:44073)
0929 01:10:17.033325 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:17.033338 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:17.033378 (+ 40us) server_negotiation.cc:407] Connection header received
0929 01:10:17.035226 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:17.035246 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:17.035618 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:17.035936 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:17.037749 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:17.039040 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:17.040990 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:17.041756 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:17.041908 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:17.046938 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:17.046947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:17.046963 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:17.047005 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:17.050610 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:17.051353 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:17.051360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:17.051366 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:17.051549 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:17.052001 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:17.052006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:17.052008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:17.052643 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:17.052996 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:17.053452 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 01:10:17.053854 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":384,"threads_started":1}
I20260929 01:10:17.061061 18300 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a8cc38804409e89fa5ac821673678" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44073 } }
I20260929 01:10:17.061504 18300 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:10:17.061612 18300 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee1a8cc38804409e89fa5ac821673678, State: Initialized, Role: FOLLOWER
I20260929 01:10:17.062201 18300 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a8cc38804409e89fa5ac821673678" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44073 } }
I20260929 01:10:17.062427 18300 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:10:17.062572 18300 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:10:17.062745 18300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:10:17.064860 18300 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a8cc38804409e89fa5ac821673678" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44073 } }
I20260929 01:10:17.065322 18300 leader_election.cc:302] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee1a8cc38804409e89fa5ac821673678; no voters:
I20260929 01:10:17.065913 18300 leader_election.cc:288] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:10:17.066118 18305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:10:17.067335 18305 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 1 LEADER]: Becoming Leader. State: Replica: ee1a8cc38804409e89fa5ac821673678, State: Running, Role: LEADER
I20260929 01:10:17.068130 18300 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:10:17.068042 18305 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a8cc38804409e89fa5ac821673678" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44073 } }
I20260929 01:10:17.071260 18307 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee1a8cc38804409e89fa5ac821673678. Latest consensus state: current_term: 1 leader_uuid: "ee1a8cc38804409e89fa5ac821673678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a8cc38804409e89fa5ac821673678" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44073 } } }
I20260929 01:10:17.071288 18306 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee1a8cc38804409e89fa5ac821673678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a8cc38804409e89fa5ac821673678" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44073 } } }
I20260929 01:10:17.071769 18306 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:17.071774 18307 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:17.073401 18311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:10:17.078524 18311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:10:17.083506 18300 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 01:10:17.094980 18311 catalog_manager.cc:1347] Generated new cluster ID: 79e66df0532f41a8b624b7820e79718e
I20260929 01:10:17.095180 18311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:10:17.117146 18311 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:10:17.119074 18311 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:10:17.130976 18311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678: Generated new TSK 0
I20260929 01:10:17.132129 18311 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:10:17.888545 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.198.212:50541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44073
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:10:18 dist-test-slave-41cl krb5kdc[17789](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790644218, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 01:10:18.291020 18345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:18.291386 18345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:18.291564 18345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:18.299921 18345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:18.300093 18345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:18.300381 18345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:10:18.313288 18345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.198.212:50541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:18.314821 18345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:18.316176 18345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:18.340075 18355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:18.340693 18356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:18.354297 18358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:18.354287 18345 server_base.cc:1034] running on GCE node
I20260929 01:10:18.355370 18345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:18.356776 18345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:18.358120 18345 hybrid_clock.cc:648] HybridClock initialized: now 1790644218358052 us; error 47 us; skew 500 ppm
I20260929 01:10:18.358579 18345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:18 dist-test-slave-41cl krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644218, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:18.363047 18345 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:10:18.365114 18345 webserver.cc:460] Webserver started at http://127.5.198.193:33191/ using document root <none> and password file <none>
I20260929 01:10:18.366048 18345 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:18.366178 18345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:18.366529 18345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:10:18 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644218, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
I20260929 01:10:18.390058 18345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "53df5158a06b4d21bcf969250316a0e9"
format_stamp: "Formatted at 2026-09-29 01:10:18 on dist-test-slave-41cl"
server_key: "5cded9f40d8ea16d1f1b5ecee4532fa6"
server_key_iv: "f367ea430e522224fcaa39f34b6513be"
server_key_version: "kuduclusterkey@0"
I20260929 01:10:18.392138 18345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "53df5158a06b4d21bcf969250316a0e9"
format_stamp: "Formatted at 2026-09-29 01:10:18 on dist-test-slave-41cl"
server_key: "5cded9f40d8ea16d1f1b5ecee4532fa6"
server_key_iv: "f367ea430e522224fcaa39f34b6513be"
server_key_version: "kuduclusterkey@0"
I20260929 01:10:18.405604 18345 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.007s
I20260929 01:10:18.433751 18365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:18.435945 18345 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.003s
I20260929 01:10:18.436188 18345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "53df5158a06b4d21bcf969250316a0e9"
format_stamp: "Formatted at 2026-09-29 01:10:18 on dist-test-slave-41cl"
server_key: "5cded9f40d8ea16d1f1b5ecee4532fa6"
server_key_iv: "f367ea430e522224fcaa39f34b6513be"
server_key_version: "kuduclusterkey@0"
I20260929 01:10:18.436650 18345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:18.468829 18345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:18.473101 18345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:18.473453 18345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:18.474570 18345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:18.476184 18345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:10:18.476274 18345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:18.476433 18345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:10:18.476547 18345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 01:10:18.561725 18345 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:36229
I20260929 01:10:18.561753 18477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:36229 every 8 connection(s)
I20260929 01:10:18.563261 18345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:10:18.570346 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 18345
I20260929 01:10:18.570573 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:10:18.571553 5915 external_mini_cluster.cc:1428] Setting key 76f4f3de27a48b47353174e4ce79058c
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
Sep 29 01:10:18 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644218, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
I20260929 01:10:18.629995 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:18.569196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44073 (local address 127.5.198.193:43311)
0929 01:10:18.570821 (+ 1625us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:18.570884 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:18.573433 (+ 2549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:18.593770 (+ 20337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:18.593788 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:18.594547 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:18.595558 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:18.595590 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:18.596956 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:18.596965 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:18.599437 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:18.599453 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:18.600953 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:18.600960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:18.601291 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:18.602484 (+ 1193us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:18.602524 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:18.608088 (+ 5564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:18.624002 (+ 15914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:18.624017 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:18.624044 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:18.624874 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:18.625415 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:18.625422 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:18.625428 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:18.625695 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:18.626421 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:18.626433 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:18.626920 (+ 487us) client_negotiation.cc:769] Sending connection context
0929 01:10:18.627214 (+ 294us) client_negotiation.cc:240] Negotiation successful
0929 01:10:18.627845 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1286,"thread_start_us":3596,"threads_started":1}
I20260929 01:10:18.633431 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:18.568261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:43311 (local address 127.5.198.254:44073)
0929 01:10:18.592949 (+ 24688us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:18.592958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:18.592990 (+ 32us) server_negotiation.cc:407] Connection header received
0929 01:10:18.593069 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:18.593077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:18.593196 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:18.595792 (+ 2596us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:18.595959 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:18.596779 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:18.599640 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:18.600401 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:18.600637 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:18.609232 (+ 8595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:18.609273 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:18.609286 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:18.609372 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:18.623372 (+ 14000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:18.625076 (+ 1704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:18.625085 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:18.625092 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:18.625271 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:18.625844 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:18.625851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:18.625857 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:18.626254 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:18.632490 (+ 6236us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:18.632681 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 01:10:18.632964 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24507,"thread_start_us":24341,"threads_started":1}
I20260929 01:10:18.635218 18478 heartbeater.cc:347] Connected to a master server at 127.5.198.254:44073
I20260929 01:10:18.635583 18478 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:18.636340 18478 heartbeater.cc:511] Master 127.5.198.254:44073 requested a full tablet report, sending...
I20260929 01:10:18.638818 18265 ts_manager.cc:194] Registered new tserver with Master: 53df5158a06b4d21bcf969250316a0e9 (127.5.198.193:36229)
I20260929 01:10:18.640527 18265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:43311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:10:18.649152 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.198.212:50541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44073
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:10:18 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644218, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
Sep 29 01:10:19 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644218, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
W20260929 01:10:19.109483 18484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:19.109994 18484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:19.110262 18484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:19.121843 18484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:19.122092 18484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:19.122334 18484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:10:19.138407 18484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.198.212:50541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:19.140450 18484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:19.142318 18484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:19.200626 18492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:19.204334 18494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:19.204862 18491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:19.205787 18484 server_base.cc:1034] running on GCE node
I20260929 01:10:19.206779 18484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:19.217841 18484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:19.222378 18484 hybrid_clock.cc:648] HybridClock initialized: now 1790644219222220 us; error 148 us; skew 500 ppm
I20260929 01:10:19.222908 18484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:19 dist-test-slave-41cl krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:19.227770 18484 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:10:19.257978 18484 webserver.cc:460] Webserver started at http://127.5.198.194:32989/ using document root <none> and password file <none>
I20260929 01:10:19.259177 18484 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:19.259408 18484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:19.259845 18484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:10:19 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
I20260929 01:10:19.297278 18484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "bf5aef212b824417b594b4beba5ee10c"
format_stamp: "Formatted at 2026-09-29 01:10:19 on dist-test-slave-41cl"
server_key: "9d0db7d34f169e26958a95c6941a1bc4"
server_key_iv: "9ef4d67c73be709e3391569ae9e9b9b4"
server_key_version: "kuduclusterkey@0"
I20260929 01:10:19.298574 18484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "bf5aef212b824417b594b4beba5ee10c"
format_stamp: "Formatted at 2026-09-29 01:10:19 on dist-test-slave-41cl"
server_key: "9d0db7d34f169e26958a95c6941a1bc4"
server_key_iv: "9ef4d67c73be709e3391569ae9e9b9b4"
server_key_version: "kuduclusterkey@0"
I20260929 01:10:19.312446 18484 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.008s sys 0.004s
I20260929 01:10:19.359764 18501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:19.361902 18484 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.006s sys 0.000s
I20260929 01:10:19.362198 18484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "bf5aef212b824417b594b4beba5ee10c"
format_stamp: "Formatted at 2026-09-29 01:10:19 on dist-test-slave-41cl"
server_key: "9d0db7d34f169e26958a95c6941a1bc4"
server_key_iv: "9ef4d67c73be709e3391569ae9e9b9b4"
server_key_version: "kuduclusterkey@0"
I20260929 01:10:19.362648 18484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:19.430728 18484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:19.439882 18484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:19.440313 18484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:19.457182 18484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:19.469494 18484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:10:19.469695 18484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 01:10:19.469869 18484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:10:19.469959 18484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:19.558951 18484 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:41661
I20260929 01:10:19.558974 18620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:41661 every 8 connection(s)
I20260929 01:10:19.560719 18484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:10:19.570420 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 18484
I20260929 01:10:19.570588 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:10:19.571126 5915 external_mini_cluster.cc:1428] Setting key b7279df9653cb40cbfa0bfecbe3031ee
Sep 29 01:10:19 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
I20260929 01:10:19.609159 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:19.569519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:37451 (local address 127.5.198.254:44073)
0929 01:10:19.587843 (+ 18324us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:19.587852 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:19.587875 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:10:19.587942 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:19.587950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:19.588054 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:19.588172 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:19.590502 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:19.591787 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:19.594512 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:19.595242 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:19.595369 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:19.600492 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:19.600504 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:19.600511 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:19.600569 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:19.604391 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:19.605298 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:19.605304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:19.605307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:19.605451 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:19.606269 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:19.606274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:19.606278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:19.606719 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:19.606880 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:19.608308 (+ 1428us) server_negotiation.cc:299] Negotiation successful
0929 01:10:19.608628 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18177,"thread_start_us":28710,"threads_started":1}
I20260929 01:10:19.611819 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:19.570377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44073 (local address 127.5.198.194:37451)
0929 01:10:19.580686 (+ 10309us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:19.580740 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:19.583569 (+ 2829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:19.588671 (+ 5102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:19.588687 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:19.589374 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:19.590271 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:19.590313 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:19.592009 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:19.592017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:19.594342 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:19.594360 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:19.595714 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:19.595723 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:19.596046 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:19.596967 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:19.597003 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:19.600139 (+ 3136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:19.604645 (+ 4506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:19.604658 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:19.604682 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:19.605147 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:19.605582 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:19.605586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:19.605590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:19.606112 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:19.606959 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:19.606971 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:19.607680 (+ 709us) client_negotiation.cc:769] Sending connection context
0929 01:10:19.609489 (+ 1809us) client_negotiation.cc:240] Negotiation successful
0929 01:10:19.610141 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9974,"thread_start_us":9727,"threads_started":1}
I20260929 01:10:19.613792 18621 heartbeater.cc:347] Connected to a master server at 127.5.198.254:44073
I20260929 01:10:19.614144 18621 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:19.615056 18621 heartbeater.cc:511] Master 127.5.198.254:44073 requested a full tablet report, sending...
I20260929 01:10:19.617089 18265 ts_manager.cc:194] Registered new tserver with Master: bf5aef212b824417b594b4beba5ee10c (127.5.198.194:41661)
I20260929 01:10:19.618019 18265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:37451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:10:19.640796 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.198.212:50541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44073
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:10:19.644327 18478 heartbeater.cc:503] Master 127.5.198.254:44073 was elected leader, sending a full tablet report...
W20260929 01:10:19.967749 18627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:19.968155 18627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:19.968256 18627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:19.975821 18627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:19.975939 18627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:19.976055 18627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:10:19.986387 18627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.198.212:50541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:44073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:19.987623 18627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:19.988935 18627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:20.002471 18635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:20.002509 18632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:20.002810 18633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:20.003175 18627 server_base.cc:1034] running on GCE node
I20260929 01:10:20.003801 18627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:20.004822 18627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:20.006054 18627 hybrid_clock.cc:648] HybridClock initialized: now 1790644220005998 us; error 50 us; skew 500 ppm
I20260929 01:10:20.006397 18627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:20 dist-test-slave-41cl krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:20.009784 18627 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:10:20.011556 18627 webserver.cc:460] Webserver started at http://127.5.198.195:37673/ using document root <none> and password file <none>
I20260929 01:10:20.012413 18627 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:20.012603 18627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:20.012943 18627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:10:20 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
I20260929 01:10:20.029234 18627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "c7d3b913306a4c0fb3c90e6482d1f84e"
format_stamp: "Formatted at 2026-09-29 01:10:20 on dist-test-slave-41cl"
server_key: "907e40cfd231afb906f0e0b499df9105"
server_key_iv: "1edccac7dff846471128c063cba7f4bf"
server_key_version: "kuduclusterkey@0"
I20260929 01:10:20.030414 18627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "c7d3b913306a4c0fb3c90e6482d1f84e"
format_stamp: "Formatted at 2026-09-29 01:10:20 on dist-test-slave-41cl"
server_key: "907e40cfd231afb906f0e0b499df9105"
server_key_iv: "1edccac7dff846471128c063cba7f4bf"
server_key_version: "kuduclusterkey@0"
I20260929 01:10:20.038125 18627 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260929 01:10:20.055223 18642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:20.057062 18627 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:10:20.057293 18627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "c7d3b913306a4c0fb3c90e6482d1f84e"
format_stamp: "Formatted at 2026-09-29 01:10:20 on dist-test-slave-41cl"
server_key: "907e40cfd231afb906f0e0b499df9105"
server_key_iv: "1edccac7dff846471128c063cba7f4bf"
server_key_version: "kuduclusterkey@0"
I20260929 01:10:20.057546 18627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:20.086244 18627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:20.090466 18627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:20.090790 18627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:20.091950 18627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:20.093747 18627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:10:20.093849 18627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:20.093976 18627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:10:20.094046 18627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:20.126567 18627 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:37763
I20260929 01:10:20.126608 18754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:37763 every 8 connection(s)
I20260929 01:10:20.127820 18627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:10:20.134555 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 18627
I20260929 01:10:20.134734 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:10:20.135221 5915 external_mini_cluster.cc:1428] Setting key ba546ae5f81b85932cdaca9eb3f5bb2f
Sep 29 01:10:20 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:20.148233 18757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:20.130480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:34889 (local address 127.5.198.254:44073)
0929 01:10:20.130934 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:20.130941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:20.131956 (+ 1015us) server_negotiation.cc:407] Connection header received
0929 01:10:20.133755 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:20.133759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:20.133838 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:20.133950 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:20.135490 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:20.136436 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:20.138179 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:20.138725 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:20.138862 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:20.142141 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:20.142146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:20.142149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:20.142209 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:20.145122 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:20.145883 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:20.145890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:20.145894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:20.146023 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:20.146466 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:20.146472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:20.146476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:20.146776 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:20.146988 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:20.147633 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 01:10:20.147866 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":203,"threads_started":1}
I20260929 01:10:20.149339 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:20.131084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44073 (local address 127.5.198.195:34889)
0929 01:10:20.131775 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:20.131814 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:20.133493 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:20.134152 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:20.134160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:20.134621 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:20.135270 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:20.135292 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:20.136642 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:20.136652 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:20.138033 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:20.138045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:20.138952 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:20.138959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:20.139193 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:20.139823 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:20.139845 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:20.141911 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:20.145279 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:20.145320 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:20.145335 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:20.145755 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:20.146134 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:20.146138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:20.146142 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:20.146354 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:20.146926 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:20.146935 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:20.147274 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 01:10:20.147556 (+ 282us) client_negotiation.cc:240] Negotiation successful
0929 01:10:20.148067 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":286,"threads_started":1}
I20260929 01:10:20.150796 18755 heartbeater.cc:347] Connected to a master server at 127.5.198.254:44073
I20260929 01:10:20.151105 18755 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:20.151829 18755 heartbeater.cc:511] Master 127.5.198.254:44073 requested a full tablet report, sending...
I20260929 01:10:20.153308 18265 ts_manager.cc:194] Registered new tserver with Master: c7d3b913306a4c0fb3c90e6482d1f84e (127.5.198.195:37763)
I20260929 01:10:20.154227 18265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:34889
I20260929 01:10:20.163601 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:10:20.165309 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 18345
Sep 29 01:10:20 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:10:20.184589 18760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:20.166383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.5.198.193:36229)
0929 01:10:20.167042 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:20.167050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:20.167077 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:10:20.167136 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:20.167142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:20.167338 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:20.167471 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:20.169202 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:20.170514 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:20.172114 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:20.172794 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:20.172948 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:20.177658 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:20.177671 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:20.177687 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:20.177725 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:20.181310 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:20.182284 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:20.182294 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:20.182296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:20.182487 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:20.182911 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:20.182915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:20.182917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:20.183375 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:20.183625 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:20.183978 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 01:10:20.184216 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":352,"threads_started":1}
I20260929 01:10:20.186054 18473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:20.271454 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 18345
I20260929 01:10:20.288427 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 18484
Sep 29 01:10:20 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:10:20.305945 18762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:20.289416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.5.198.194:41661)
0929 01:10:20.290132 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:20.290139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:20.290168 (+ 29us) server_negotiation.cc:407] Connection header received
0929 01:10:20.290235 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:20.290241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:20.290459 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:20.290615 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:20.292047 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:20.293327 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:20.294901 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:20.295477 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:20.295580 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:20.300212 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:20.300225 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:20.300240 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:20.300275 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:20.303275 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:20.303952 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:20.303959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:20.303961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:20.304138 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:20.304518 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:20.304526 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:20.304529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:20.304870 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:20.305122 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:20.305466 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 01:10:20.305659 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":309,"threads_started":1}
I20260929 01:10:20.307209 18616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:20.393066 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 18484
I20260929 01:10:20.409510 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 18627
Sep 29 01:10:20 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:10:20.425783 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:20.410446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40638 (local address 127.5.198.195:37763)
0929 01:10:20.411019 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:20.411026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:20.411054 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:10:20.411112 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:20.411118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:20.411323 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:20.411490 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:20.412851 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:20.413842 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:20.415127 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:20.415678 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:20.415821 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:20.420118 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:20.420129 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:20.420144 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:20.420175 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:20.422953 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:20.423686 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:20.423697 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:20.423700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:20.423878 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:20.424293 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:20.424301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:20.424307 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:20.424806 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:20.425093 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:20.425296 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 01:10:20.425487 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":264,"threads_started":1}
I20260929 01:10:20.427066 18750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:20.509725 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 18627
I20260929 01:10:20.525990 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 18233
I20260929 01:10:20.527305 18295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:20.634459 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 18233
I20260929 01:10:20.659466 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:44073
--webserver_interface=127.5.198.254
--webserver_port=38895
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:44073
--ranger_config_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.198.212:50541/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.198.212:50541/kms with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:10:20.982525 18767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:20.982865 18767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:20.983033 18767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:20.990211 18767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:10:20.990375 18767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:20.990490 18767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:20.990599 18767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:10:20.990688 18767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:10:21.000797 18767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.198.212:50541/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:44073
--ranger_config_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:44073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=38895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:21.002169 18767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:21.003557 18767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:21.015470 18775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:21.015524 18772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:21.015563 18773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:21.016919 18767 server_base.cc:1034] running on GCE node
I20260929 01:10:21.017721 18767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:21.019232 18767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:21.020520 18767 hybrid_clock.cc:648] HybridClock initialized: now 1790644221020431 us; error 38 us; skew 500 ppm
I20260929 01:10:21.020900 18767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:21 dist-test-slave-41cl krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644221, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:21.024051 18767 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:10:21.025540 18767 webserver.cc:460] Webserver started at http://127.5.198.254:38895/ using document root <none> and password file <none>
I20260929 01:10:21.026149 18767 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:21.026249 18767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 01:10:21 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644221, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
I20260929 01:10:21.091516 18767 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:10:21.094770 18783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:21.096364 18767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:10:21.096616 18767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "ee1a8cc38804409e89fa5ac821673678"
format_stamp: "Formatted at 2026-09-29 01:10:16 on dist-test-slave-41cl"
server_key: "90c039075ca7fb91affd1ff25d2deba8"
server_key_iv: "9b8d085b7f5cef52328bcaab5d4bf733"
server_key_version: "kuduclusterkey@0"
I20260929 01:10:21.097568 18767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:21.130844 18767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:21.135004 18767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:21.135313 18767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:21.154419 18767 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:44073
I20260929 01:10:21.154454 18834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:44073 every 8 connection(s)
I20260929 01:10:21.155900 18767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:10:21.162794 18835 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:10:21.163249 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 18767
I20260929 01:10:21.164113 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.198.212:50541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:36229
--local_ip_for_outbound_sockets=127.5.198.193
--tserver_master_addrs=127.5.198.254:44073
--webserver_port=33191
--webserver_interface=127.5.198.193
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.198.212:50541/kms with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:10:21.166198 18835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678: Bootstrap starting.
I20260929 01:10:21.184490 18835 log.cc:824] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:21.193342 18835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:10:21.193882 18835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678: Bootstrap complete.
I20260929 01:10:21.203117 18835 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a8cc38804409e89fa5ac821673678" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44073 } }
I20260929 01:10:21.203999 18835 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee1a8cc38804409e89fa5ac821673678, State: Initialized, Role: FOLLOWER
I20260929 01:10:21.204681 18835 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a8cc38804409e89fa5ac821673678" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44073 } }
I20260929 01:10:21.204909 18835 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:10:21.205032 18835 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:10:21.205204 18835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:10:21.209203 18835 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a8cc38804409e89fa5ac821673678" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44073 } }
I20260929 01:10:21.209705 18835 leader_election.cc:302] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee1a8cc38804409e89fa5ac821673678; no voters:
I20260929 01:10:21.210323 18835 leader_election.cc:288] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 01:10:21.210457 18839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:10:21.211727 18839 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [term 2 LEADER]: Becoming Leader. State: Replica: ee1a8cc38804409e89fa5ac821673678, State: Running, Role: LEADER
I20260929 01:10:21.212345 18839 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a8cc38804409e89fa5ac821673678" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44073 } }
I20260929 01:10:21.212702 18835 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:10:21.214949 18840 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ee1a8cc38804409e89fa5ac821673678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a8cc38804409e89fa5ac821673678" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44073 } } }
I20260929 01:10:21.215387 18840 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:21.217121 18844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:10:21.217031 18841 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee1a8cc38804409e89fa5ac821673678. Latest consensus state: current_term: 2 leader_uuid: "ee1a8cc38804409e89fa5ac821673678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a8cc38804409e89fa5ac821673678" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44073 } } }
I20260929 01:10:21.217519 18841 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:21.223300 18844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:10:21.224746 18844 catalog_manager.cc:1259] Loaded cluster ID: 79e66df0532f41a8b624b7820e79718e
I20260929 01:10:21.224888 18844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:10:21.228612 18844 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:10:21.229954 18844 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678: Loaded TSK: 0
I20260929 01:10:21.231411 18844 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:10:21.234642 18835 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260929 01:10:21.575310 18837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:21.575624 18837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:21.575805 18837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:21.585813 18837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:21.586066 18837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:21.586315 18837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:10:21.602352 18837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.198.212:50541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:36229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=33191
--tserver_master_addrs=127.5.198.254:44073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:21.604481 18837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:21.606673 18837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:21.644693 18873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:21.647210 18872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:21.647787 18875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:21.648485 18837 server_base.cc:1034] running on GCE node
I20260929 01:10:21.649305 18837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:21.662443 18837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:21.663821 18837 hybrid_clock.cc:648] HybridClock initialized: now 1790644221663686 us; error 111 us; skew 500 ppm
I20260929 01:10:21.664278 18837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:21 dist-test-slave-41cl krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644221, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:21.667882 18837 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:10:21.677927 18837 webserver.cc:460] Webserver started at http://127.5.198.193:33191/ using document root <none> and password file <none>
I20260929 01:10:21.678905 18837 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:21.679126 18837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 01:10:21 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644221, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
I20260929 01:10:21.718611 18837 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 01:10:21.728847 18883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:21.730872 18837 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.005s
I20260929 01:10:21.731142 18837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "53df5158a06b4d21bcf969250316a0e9"
format_stamp: "Formatted at 2026-09-29 01:10:18 on dist-test-slave-41cl"
server_key: "5cded9f40d8ea16d1f1b5ecee4532fa6"
server_key_iv: "f367ea430e522224fcaa39f34b6513be"
server_key_version: "kuduclusterkey@0"
I20260929 01:10:21.732729 18837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:21.757505 18837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:21.762974 18837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:21.763350 18837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:21.770676 18837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:21.778918 18837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:10:21.779111 18837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:21.779294 18837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:10:21.779428 18837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:21.899655 18837 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:36229
I20260929 01:10:21.901719 18837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:10:21.899699 18996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:36229 every 8 connection(s)
I20260929 01:10:21.906477 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 18837
I20260929 01:10:21.907521 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.198.212:50541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:41661
--local_ip_for_outbound_sockets=127.5.198.194
--tserver_master_addrs=127.5.198.254:44073
--webserver_port=32989
--webserver_interface=127.5.198.194
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.198.212:50541/kms with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:10:21 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644221, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:21.967350 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:21.905464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44073 (local address 127.5.198.193:47509)
0929 01:10:21.937216 (+ 31752us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:21.937275 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:21.941192 (+ 3917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:21.944846 (+ 3654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:21.944860 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:21.945567 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:21.946474 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:21.946505 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:21.949346 (+ 2841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:21.949353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:21.951950 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:21.951968 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:21.953826 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:21.953833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:21.954101 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:21.954952 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:21.954982 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:21.957778 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:21.962352 (+ 4574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:21.962361 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:21.962379 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:21.962984 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:21.963523 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:21.963528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:21.963532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:21.963772 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:21.964871 (+ 1099us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:21.964880 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:21.965268 (+ 388us) client_negotiation.cc:769] Sending connection context
0929 01:10:21.965552 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 01:10:21.966089 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31416,"thread_start_us":31187,"threads_started":1}
I20260929 01:10:21.969655 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:21.905299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:47509 (local address 127.5.198.254:44073)
0929 01:10:21.921025 (+ 15726us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:21.921055 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:21.937802 (+ 16747us) server_negotiation.cc:407] Connection header received
0929 01:10:21.942038 (+ 4236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:21.942059 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:21.942602 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:21.943042 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:21.946754 (+ 3712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:21.949112 (+ 2358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:21.952164 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:21.953269 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:21.953439 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:21.958118 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:21.958133 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:21.958154 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:21.958202 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:21.962177 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:21.963165 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:21.963175 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:21.963183 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:21.963384 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:21.963919 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:21.963924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:21.963928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:21.964689 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:21.964883 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:21.968069 (+ 3186us) server_negotiation.cc:299] Negotiation successful
0929 01:10:21.968512 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14993,"thread_start_us":15029,"threads_started":1}
I20260929 01:10:21.973798 18997 heartbeater.cc:347] Connected to a master server at 127.5.198.254:44073
I20260929 01:10:21.974123 18997 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:21.974963 18997 heartbeater.cc:511] Master 127.5.198.254:44073 requested a full tablet report, sending...
W20260929 01:10:21.978055 18997 heartbeater.cc:644] Failed to heartbeat to 127.5.198.254:44073 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:10:21.978286 18997 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:21.978554 18997 heartbeater.cc:511] Master 127.5.198.254:44073 requested a full tablet report, sending...
I20260929 01:10:21.979851 18997 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:21.980127 18997 heartbeater.cc:511] Master 127.5.198.254:44073 requested a full tablet report, sending...
W20260929 01:10:21.981361 18997 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:10:22.329797 19001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:22.330068 19001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:22.330211 19001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:22.340639 19001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:22.340842 19001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:22.341054 19001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:10:22.356407 19001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.198.212:50541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:41661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=32989
--tserver_master_addrs=127.5.198.254:44073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:22.358271 19001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:22.360038 19001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:22.398172 19018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:22.401448 19016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:22.412011 19001 server_base.cc:1034] running on GCE node
W20260929 01:10:22.413530 19020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:22.421329 19001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:22.422705 19001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:22.424013 19001 hybrid_clock.cc:648] HybridClock initialized: now 1790644222423945 us; error 57 us; skew 500 ppm
I20260929 01:10:22.424412 19001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:22 dist-test-slave-41cl krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644222, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:22.428913 19001 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:10:22.434073 19001 webserver.cc:460] Webserver started at http://127.5.198.194:32989/ using document root <none> and password file <none>
I20260929 01:10:22.435065 19001 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:22.435232 19001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 01:10:22 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644222, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
I20260929 01:10:22.471542 19001 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.004s sys 0.004s
Sep 29 01:10:22 dist-test-slave-41cl krb5kdc[17789](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790644222, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:22.493180 19028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:22.495357 19001 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.004s sys 0.015s
I20260929 01:10:22.495546 19001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "bf5aef212b824417b594b4beba5ee10c"
format_stamp: "Formatted at 2026-09-29 01:10:19 on dist-test-slave-41cl"
server_key: "9d0db7d34f169e26958a95c6941a1bc4"
server_key_iv: "9ef4d67c73be709e3391569ae9e9b9b4"
server_key_version: "kuduclusterkey@0"
I20260929 01:10:22.496984 19001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:22.543998 19001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:22.549695 19001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:22.550119 19001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:22.551463 19001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:22.559425 19001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:10:22.559616 19001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 01:10:22.559791 19001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:10:22.559935 19001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:22.654596 19001 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:41661
I20260929 01:10:22.656606 19001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:10:22.654633 19141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:41661 every 8 connection(s)
I20260929 01:10:22.664083 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 19001
I20260929 01:10:22.665031 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.198.212:50541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:37763
--local_ip_for_outbound_sockets=127.5.198.195
--tserver_master_addrs=127.5.198.254:44073
--webserver_port=37673
--webserver_interface=127.5.198.195
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.198.212:50541/kms with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:10:22 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644222, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:22.725773 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:22.661358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:41827 (local address 127.5.198.254:44073)
0929 01:10:22.692871 (+ 31513us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:22.692879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:22.692909 (+ 30us) server_negotiation.cc:407] Connection header received
0929 01:10:22.703220 (+ 10311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:22.703227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:22.703328 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:22.703414 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:22.708661 (+ 5247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:22.709816 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:22.712309 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:22.712983 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:22.713087 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:22.718413 (+ 5326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:22.718421 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:22.718426 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:22.718470 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:22.722049 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:22.723005 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:22.723012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:22.723016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:22.723157 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:22.723742 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:22.723748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:22.723752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:22.724124 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:22.724240 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:22.725160 (+ 920us) server_negotiation.cc:299] Negotiation successful
0929 01:10:22.725392 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31344,"thread_start_us":36091,"threads_started":1}
I20260929 01:10:22.728073 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:22.660755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44073 (local address 127.5.198.194:41827)
0929 01:10:22.689980 (+ 29225us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:22.690066 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:22.699926 (+ 9860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:22.705548 (+ 5622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:22.705561 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:22.706266 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:22.707154 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:22.707180 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:22.710041 (+ 2861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:22.710047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:22.712131 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:22.712147 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:22.713434 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:22.713441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:22.713705 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:22.714526 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:22.714555 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:22.718143 (+ 3588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:22.722226 (+ 4083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:22.722235 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:22.722250 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:22.722864 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:22.723329 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:22.723334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:22.723338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:22.723614 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:22.724339 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:22.724346 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:22.724740 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 01:10:22.726350 (+ 1610us) client_negotiation.cc:240] Negotiation successful
0929 01:10:22.726867 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28783,"thread_start_us":32070,"threads_started":1}
I20260929 01:10:22.729842 19142 heartbeater.cc:347] Connected to a master server at 127.5.198.254:44073
I20260929 01:10:22.730172 19142 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:22.731122 19142 heartbeater.cc:511] Master 127.5.198.254:44073 requested a full tablet report, sending...
I20260929 01:10:22.733189 18800 ts_manager.cc:194] Registered new tserver with Master: bf5aef212b824417b594b4beba5ee10c (127.5.198.194:41661)
I20260929 01:10:22.735358 18800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:41827
I20260929 01:10:22.981539 18997 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:22.982015 18997 heartbeater.cc:511] Master 127.5.198.254:44073 requested a full tablet report, sending...
I20260929 01:10:22.983485 18800 ts_manager.cc:194] Registered new tserver with Master: 53df5158a06b4d21bcf969250316a0e9 (127.5.198.193:36229)
I20260929 01:10:22.984941 18800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:47509
Sep 29 01:10:23 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644222, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
W20260929 01:10:23.118000 19146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:23.118345 19146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:23.118496 19146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:23.126114 19146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:23.126268 19146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:23.126457 19146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:10:23.136281 19146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.198.212:50541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:37763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=37673
--tserver_master_addrs=127.5.198.254:44073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:23.137838 19146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:23.139532 19146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:23.163721 19151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:23.166057 19152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:23.173324 19154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:23.173952 19146 server_base.cc:1034] running on GCE node
I20260929 01:10:23.174629 19146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:23.175693 19146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:23.177006 19146 hybrid_clock.cc:648] HybridClock initialized: now 1790644223176932 us; error 55 us; skew 500 ppm
I20260929 01:10:23.177428 19146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:23 dist-test-slave-41cl krb5kdc[17789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644223, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:23.180991 19146 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:10:23.182948 19146 webserver.cc:460] Webserver started at http://127.5.198.195:37673/ using document root <none> and password file <none>
I20260929 01:10:23.183709 19146 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:23.183887 19146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 01:10:23 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644223, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
I20260929 01:10:23.210310 19146 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.005s sys 0.003s
I20260929 01:10:23.213721 19162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:23.215444 19146 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:10:23.215648 19146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "c7d3b913306a4c0fb3c90e6482d1f84e"
format_stamp: "Formatted at 2026-09-29 01:10:20 on dist-test-slave-41cl"
server_key: "907e40cfd231afb906f0e0b499df9105"
server_key_iv: "1edccac7dff846471128c063cba7f4bf"
server_key_version: "kuduclusterkey@0"
I20260929 01:10:23.216805 19146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:23.234107 19146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:23.238374 19146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:23.238663 19146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:23.239882 19146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:23.249385 19146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:10:23.249517 19146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:23.249676 19146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:10:23.249766 19146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:23.308924 19146 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:37763
I20260929 01:10:23.309003 19274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:37763 every 8 connection(s)
I20260929 01:10:23.310990 19146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:10:23.320660 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 19146
Sep 29 01:10:23 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644222, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for HTTP/127.5.198.212@KRBTEST.COM
Sep 29 01:10:23 dist-test-slave-41cl krb5kdc[17789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644223, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:23.402241 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:23.354064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57104 (local address 127.5.198.254:44073)
0929 01:10:23.360784 (+ 6720us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:23.360790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:23.361709 (+ 919us) server_negotiation.cc:407] Connection header received
0929 01:10:23.361775 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:23.361780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:23.361878 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:23.361982 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:23.367474 (+ 5492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.368662 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.374356 (+ 5694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.374997 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.375100 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:23.382775 (+ 7675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:23.382783 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:23.382788 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:23.382831 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:23.386348 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:23.389101 (+ 2753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:23.389107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:23.389110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:23.389285 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:23.397668 (+ 8383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:23.397675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:23.397678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:23.398073 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:23.398370 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:23.401062 (+ 2692us) server_negotiation.cc:299] Negotiation successful
0929 01:10:23.401777 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6582,"thread_start_us":6732,"threads_started":1}
I20260929 01:10:23.409490 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:23.316920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:60825 (local address 127.5.198.254:44073)
0929 01:10:23.353827 (+ 36907us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:23.353836 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:23.359862 (+ 6026us) server_negotiation.cc:407] Connection header received
0929 01:10:23.359954 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:23.359961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:23.360075 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:23.360199 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:23.370649 (+ 10450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.371848 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.378939 (+ 7091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.379656 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.379759 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:23.389884 (+ 10125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:23.389891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:23.389896 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:23.389934 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:23.393683 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:23.405855 (+ 12172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:23.405862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:23.405865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:23.406057 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:23.407015 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:23.407021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:23.407024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:23.407369 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:23.407502 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:23.408862 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0929 01:10:23.409120 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36713,"thread_start_us":36937,"threads_started":1}
I20260929 01:10:23.411755 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:23.315573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44073 (local address 127.5.198.195:60825)
0929 01:10:23.355724 (+ 40151us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:23.355790 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:23.358542 (+ 2752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:23.364733 (+ 6191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:23.364747 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:23.365582 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:23.366518 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:23.366548 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.372080 (+ 5532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.372085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:23.374139 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:23.374156 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.380149 (+ 5993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.380157 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:23.380507 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:23.381312 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:23.381346 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:23.388865 (+ 7519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:23.397249 (+ 8384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:23.397260 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:23.397293 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:23.400745 (+ 3452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:23.406347 (+ 5602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:23.406354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:23.406358 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:23.406677 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:23.407601 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:23.407610 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:23.408384 (+ 774us) client_negotiation.cc:769] Sending connection context
0929 01:10:23.409689 (+ 1305us) client_negotiation.cc:240] Negotiation successful
0929 01:10:23.410300 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39789,"thread_start_us":43317,"threads_started":1}
I20260929 01:10:23.413607 19276 heartbeater.cc:347] Connected to a master server at 127.5.198.254:44073
I20260929 01:10:23.413964 19276 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:23.414949 19276 heartbeater.cc:511] Master 127.5.198.254:44073 requested a full tablet report, sending...
I20260929 01:10:23.416711 18800 ts_manager.cc:194] Registered new tserver with Master: c7d3b913306a4c0fb3c90e6482d1f84e (127.5.198.195:37763)
I20260929 01:10:23.417937 18800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:60825
I20260929 01:10:23.422062 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:10:23.456918 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:23.432307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.5.198.254:44073)
0929 01:10:23.432619 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:23.432626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:23.438952 (+ 6326us) server_negotiation.cc:407] Connection header received
0929 01:10:23.439542 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:23.439549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:23.439640 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:23.439720 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:23.442002 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.443101 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.445176 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.445848 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.445954 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:23.449604 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:23.449612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:23.449617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:23.449663 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:23.453244 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:23.454176 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:23.454183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:23.454187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:23.454359 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:23.454871 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:23.454877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:23.454881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:23.455332 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:23.455578 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:23.456193 (+ 615us) server_negotiation.cc:299] Negotiation successful
0929 01:10:23.456486 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:10:23.464331 18800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:10:23.475855 18800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:10:23.577095 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:23.521021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:41661 (local address 127.0.0.1:34260)
0929 01:10:23.560111 (+ 39090us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:23.560188 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:23.562243 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:23.564069 (+ 1826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:23.564082 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:23.564131 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:10:23.564768 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:23.564785 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.571031 (+ 6246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.571039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:23.573707 (+ 2668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:23.573723 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.576029 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.576037 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:23.576255 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:23.576329 (+ 74us) client_negotiation.cc:769] Sending connection context
0929 01:10:23.576485 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 01:10:23.576697 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38649,"threads_started":1}
I20260929 01:10:23.579897 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:23.521181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.5.198.194:41661)
0929 01:10:23.562954 (+ 41773us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:23.562965 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:23.563005 (+ 40us) server_negotiation.cc:407] Connection header received
0929 01:10:23.563133 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:23.563145 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:23.563480 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:23.563709 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:10:23.569239 (+ 5530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.570844 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.573930 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.575503 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.575639 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:23.575852 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:23.579393 (+ 3541us) server_negotiation.cc:299] Negotiation successful
0929 01:10:23.579578 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41521,"thread_start_us":47763,"threads_started":1}
I20260929 01:10:23.584299 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:23.522336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.5.198.193:36229)
0929 01:10:23.546008 (+ 23672us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:23.546024 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:23.569201 (+ 23177us) server_negotiation.cc:407] Connection header received
0929 01:10:23.569411 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:23.569422 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:23.569691 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:23.569944 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:10:23.570906 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.572581 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.581144 (+ 8563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.582688 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.582908 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:23.583164 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:23.583808 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 01:10:23.583975 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23363,"thread_start_us":23681,"threads_started":1}
I20260929 01:10:23.584138 19077 tablet_service.cc:1467] Processing CreateTablet for tablet 2c18507df4074ab98c9513f35941d7b1 (DEFAULT_TABLE table=test-table [id=e12da2e492ac4ae0a0270ac07affce86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:10:23.584972 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:23.521674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:36229 (local address 127.0.0.1:48052)
0929 01:10:23.569043 (+ 47369us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:23.569071 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:23.569240 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:23.570200 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:23.570205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:23.570234 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:10:23.570658 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:23.570670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.572827 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.572833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:23.575032 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:23.575049 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.583393 (+ 8344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.583399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:23.583597 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:23.583633 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 01:10:23.584525 (+ 892us) client_negotiation.cc:240] Negotiation successful
0929 01:10:23.584660 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47173,"thread_start_us":46994,"threads_started":1}
I20260929 01:10:23.586293 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c18507df4074ab98c9513f35941d7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:10:23.588709 18932 tablet_service.cc:1467] Processing CreateTablet for tablet 2c18507df4074ab98c9513f35941d7b1 (DEFAULT_TABLE table=test-table [id=e12da2e492ac4ae0a0270ac07affce86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:10:23.589715 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:23.525288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.5.198.195:37763)
0929 01:10:23.561061 (+ 35773us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:23.561074 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:23.577492 (+ 16418us) server_negotiation.cc:407] Connection header received
0929 01:10:23.578134 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:23.578144 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:23.578418 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:23.578586 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:10:23.579357 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.580986 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.585601 (+ 4615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.587034 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.587142 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:23.587370 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:23.589181 (+ 1811us) server_negotiation.cc:299] Negotiation successful
0929 01:10:23.589362 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35508,"thread_start_us":35779,"threads_started":1}
I20260929 01:10:23.590175 18932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c18507df4074ab98c9513f35941d7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:10:23.590556 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:23.524528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:37763 (local address 127.0.0.1:56554)
0929 01:10:23.577363 (+ 52835us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:23.577385 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:23.577598 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:23.578977 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:23.578983 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:23.579010 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:10:23.579173 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:23.579183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.581845 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.581853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:23.583744 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:23.583760 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.587581 (+ 3821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.587586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:23.588979 (+ 1393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:23.589012 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:10:23.590068 (+ 1056us) client_negotiation.cc:240] Negotiation successful
0929 01:10:23.590275 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52722,"thread_start_us":36807,"threads_started":1}
I20260929 01:10:23.593932 19210 tablet_service.cc:1467] Processing CreateTablet for tablet 2c18507df4074ab98c9513f35941d7b1 (DEFAULT_TABLE table=test-table [id=e12da2e492ac4ae0a0270ac07affce86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:10:23.595074 19210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c18507df4074ab98c9513f35941d7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:10:23.621299 19297 tablet_bootstrap.cc:492] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9: Bootstrap starting.
I20260929 01:10:23.626070 19297 tablet_bootstrap.cc:654] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9: Neither blocks nor log segments found. Creating new log.
I20260929 01:10:23.627370 19297 log.cc:824] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:23.629997 19299 tablet_bootstrap.cc:492] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c: Bootstrap starting.
I20260929 01:10:23.632989 19299 tablet_bootstrap.cc:654] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c: Neither blocks nor log segments found. Creating new log.
I20260929 01:10:23.634271 19299 log.cc:824] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:23.636322 19298 tablet_bootstrap.cc:492] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e: Bootstrap starting.
I20260929 01:10:23.636618 19297 tablet_bootstrap.cc:492] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9: No bootstrap required, opened a new log
I20260929 01:10:23.636902 19297 ts_tablet_manager.cc:1397] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.010s
I20260929 01:10:23.643707 19298 tablet_bootstrap.cc:654] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e: Neither blocks nor log segments found. Creating new log.
I20260929 01:10:23.645701 19298 log.cc:824] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:23.647382 19297 raft_consensus.cc:348] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5aef212b824417b594b4beba5ee10c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 41661 } } peers { permanent_uuid: "53df5158a06b4d21bcf969250316a0e9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36229 } } peers { permanent_uuid: "c7d3b913306a4c0fb3c90e6482d1f84e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37763 } }
I20260929 01:10:23.647837 19297 raft_consensus.cc:374] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:10:23.647928 19297 raft_consensus.cc:729] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53df5158a06b4d21bcf969250316a0e9, State: Initialized, Role: FOLLOWER
I20260929 01:10:23.648974 19297 consensus_queue.cc:260] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5aef212b824417b594b4beba5ee10c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 41661 } } peers { permanent_uuid: "53df5158a06b4d21bcf969250316a0e9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36229 } } peers { permanent_uuid: "c7d3b913306a4c0fb3c90e6482d1f84e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37763 } }
I20260929 01:10:23.654928 19297 ts_tablet_manager.cc:1428] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9: Time spent starting tablet: real 0.018s user 0.006s sys 0.010s
I20260929 01:10:23.656245 18997 heartbeater.cc:503] Master 127.5.198.254:44073 was elected leader, sending a full tablet report...
I20260929 01:10:23.656913 19299 tablet_bootstrap.cc:492] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c: No bootstrap required, opened a new log
I20260929 01:10:23.657248 19299 ts_tablet_manager.cc:1397] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.004s
I20260929 01:10:23.661489 19298 tablet_bootstrap.cc:492] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e: No bootstrap required, opened a new log
I20260929 01:10:23.661820 19298 ts_tablet_manager.cc:1397] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.008s
I20260929 01:10:23.670323 19299 raft_consensus.cc:348] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5aef212b824417b594b4beba5ee10c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 41661 } } peers { permanent_uuid: "53df5158a06b4d21bcf969250316a0e9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36229 } } peers { permanent_uuid: "c7d3b913306a4c0fb3c90e6482d1f84e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37763 } }
I20260929 01:10:23.670800 19299 raft_consensus.cc:374] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:10:23.670915 19299 raft_consensus.cc:729] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5aef212b824417b594b4beba5ee10c, State: Initialized, Role: FOLLOWER
I20260929 01:10:23.671674 19299 consensus_queue.cc:260] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5aef212b824417b594b4beba5ee10c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 41661 } } peers { permanent_uuid: "53df5158a06b4d21bcf969250316a0e9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36229 } } peers { permanent_uuid: "c7d3b913306a4c0fb3c90e6482d1f84e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37763 } }
I20260929 01:10:23.672521 19298 raft_consensus.cc:348] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5aef212b824417b594b4beba5ee10c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 41661 } } peers { permanent_uuid: "53df5158a06b4d21bcf969250316a0e9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36229 } } peers { permanent_uuid: "c7d3b913306a4c0fb3c90e6482d1f84e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37763 } }
I20260929 01:10:23.672895 19298 raft_consensus.cc:374] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:10:23.673019 19298 raft_consensus.cc:729] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7d3b913306a4c0fb3c90e6482d1f84e, State: Initialized, Role: FOLLOWER
I20260929 01:10:23.673595 19298 consensus_queue.cc:260] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5aef212b824417b594b4beba5ee10c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 41661 } } peers { permanent_uuid: "53df5158a06b4d21bcf969250316a0e9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36229 } } peers { permanent_uuid: "c7d3b913306a4c0fb3c90e6482d1f84e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37763 } }
I20260929 01:10:23.677471 19142 heartbeater.cc:503] Master 127.5.198.254:44073 was elected leader, sending a full tablet report...
I20260929 01:10:23.677994 19299 ts_tablet_manager.cc:1428] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c: Time spent starting tablet: real 0.021s user 0.009s sys 0.008s
I20260929 01:10:23.678894 19276 heartbeater.cc:503] Master 127.5.198.254:44073 was elected leader, sending a full tablet report...
I20260929 01:10:23.679618 19298 ts_tablet_manager.cc:1428] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
W20260929 01:10:23.694528 18998 tablet.cc:2369] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:10:23.708829 19143 tablet.cc:2369] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:10:23.714465 19305 raft_consensus.cc:482] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:10:23.714840 19305 raft_consensus.cc:504] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5aef212b824417b594b4beba5ee10c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 41661 } } peers { permanent_uuid: "53df5158a06b4d21bcf969250316a0e9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36229 } } peers { permanent_uuid: "c7d3b913306a4c0fb3c90e6482d1f84e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37763 } }
I20260929 01:10:23.716833 19305 leader_election.cc:288] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf5aef212b824417b594b4beba5ee10c (127.5.198.194:41661), c7d3b913306a4c0fb3c90e6482d1f84e (127.5.198.195:37763)
I20260929 01:10:23.725536 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:23.717357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:60615 (local address 127.5.198.195:37763)
0929 01:10:23.717583 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:23.717595 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:23.718264 (+ 669us) server_negotiation.cc:407] Connection header received
0929 01:10:23.718552 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:23.718556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:23.718615 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:23.718714 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:10:23.719639 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.720651 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.723406 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.724537 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.724692 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:23.724763 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:23.725156 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 01:10:23.725283 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 01:10:23.725638 19314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:23.717083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:37763 (local address 127.5.198.193:60615)
0929 01:10:23.718150 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:23.718172 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:23.718387 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:23.718904 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:23.718908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:23.718932 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:10:23.719395 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:23.719411 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.720795 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.720804 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:23.723248 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:23.723263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.724749 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.724755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:23.724938 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:23.725016 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 01:10:23.725113 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 01:10:23.725242 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":1,"thread_start_us":1027,"threads_started":1}
I20260929 01:10:23.726881 19230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c18507df4074ab98c9513f35941d7b1" candidate_uuid: "53df5158a06b4d21bcf969250316a0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7d3b913306a4c0fb3c90e6482d1f84e" is_pre_election: true
I20260929 01:10:23.727056 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:23.717754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:40653 (local address 127.5.198.194:41661)
0929 01:10:23.717973 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:23.717979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:23.718504 (+ 525us) server_negotiation.cc:407] Connection header received
0929 01:10:23.718746 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:23.718751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:23.718824 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:23.718909 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:10:23.719874 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.720927 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.724610 (+ 3683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.725989 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.726151 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:23.726221 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:23.726670 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 01:10:23.726796 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 01:10:23.727111 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:23.717203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:41661 (local address 127.5.198.193:40653)
0929 01:10:23.718405 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:23.718428 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:23.718596 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:23.719202 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:23.719206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:23.719234 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:10:23.719687 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:23.719702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.721097 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.721112 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:23.723277 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:23.723285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.726355 (+ 3070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.726363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:23.726511 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:23.726533 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:10:23.726634 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 01:10:23.726751 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"mutex_wait_us":31,"thread_start_us":548,"threads_started":1}
I20260929 01:10:23.727404 19230 raft_consensus.cc:2454] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53df5158a06b4d21bcf969250316a0e9 in term 0.
I20260929 01:10:23.728053 18887 leader_election.cc:302] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53df5158a06b4d21bcf969250316a0e9, c7d3b913306a4c0fb3c90e6482d1f84e; no voters:
I20260929 01:10:23.728219 19097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c18507df4074ab98c9513f35941d7b1" candidate_uuid: "53df5158a06b4d21bcf969250316a0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5aef212b824417b594b4beba5ee10c" is_pre_election: true
I20260929 01:10:23.728523 19305 raft_consensus.cc:2781] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:10:23.728665 19305 raft_consensus.cc:482] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:10:23.728760 19305 raft_consensus.cc:3037] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:10:23.728767 19097 raft_consensus.cc:2454] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53df5158a06b4d21bcf969250316a0e9 in term 0.
I20260929 01:10:23.731801 19305 raft_consensus.cc:504] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5aef212b824417b594b4beba5ee10c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 41661 } } peers { permanent_uuid: "53df5158a06b4d21bcf969250316a0e9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36229 } } peers { permanent_uuid: "c7d3b913306a4c0fb3c90e6482d1f84e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37763 } }
I20260929 01:10:23.732504 19305 leader_election.cc:288] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [CANDIDATE]: Term 1 election: Requested vote from peers bf5aef212b824417b594b4beba5ee10c (127.5.198.194:41661), c7d3b913306a4c0fb3c90e6482d1f84e (127.5.198.195:37763)
I20260929 01:10:23.732944 19097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c18507df4074ab98c9513f35941d7b1" candidate_uuid: "53df5158a06b4d21bcf969250316a0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5aef212b824417b594b4beba5ee10c"
I20260929 01:10:23.733035 19230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c18507df4074ab98c9513f35941d7b1" candidate_uuid: "53df5158a06b4d21bcf969250316a0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7d3b913306a4c0fb3c90e6482d1f84e"
I20260929 01:10:23.733156 19097 raft_consensus.cc:3037] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:10:23.733253 19230 raft_consensus.cc:3037] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:10:23.735483 19230 raft_consensus.cc:2454] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53df5158a06b4d21bcf969250316a0e9 in term 1.
I20260929 01:10:23.735872 18887 leader_election.cc:302] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53df5158a06b4d21bcf969250316a0e9, c7d3b913306a4c0fb3c90e6482d1f84e; no voters:
I20260929 01:10:23.736171 19305 raft_consensus.cc:2781] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:10:23.736169 19097 raft_consensus.cc:2454] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53df5158a06b4d21bcf969250316a0e9 in term 1.
I20260929 01:10:23.737177 19305 raft_consensus.cc:686] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [term 1 LEADER]: Becoming Leader. State: Replica: 53df5158a06b4d21bcf969250316a0e9, State: Running, Role: LEADER
I20260929 01:10:23.738093 19305 consensus_queue.cc:237] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5aef212b824417b594b4beba5ee10c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 41661 } } peers { permanent_uuid: "53df5158a06b4d21bcf969250316a0e9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36229 } } peers { permanent_uuid: "c7d3b913306a4c0fb3c90e6482d1f84e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37763 } }
I20260929 01:10:23.743299 18800 catalog_manager.cc:5495] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53df5158a06b4d21bcf969250316a0e9 (127.5.198.193). New cstate: current_term: 1 leader_uuid: "53df5158a06b4d21bcf969250316a0e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5aef212b824417b594b4beba5ee10c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 41661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53df5158a06b4d21bcf969250316a0e9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 36229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7d3b913306a4c0fb3c90e6482d1f84e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37763 } health_report { overall_health: UNKNOWN } } }
I20260929 01:10:23.787688 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:23.781940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48054 (local address 127.5.198.193:36229)
0929 01:10:23.782168 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:23.782176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:23.782266 (+ 90us) server_negotiation.cc:407] Connection header received
0929 01:10:23.782540 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:23.782545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:23.782634 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:23.782791 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:10:23.783180 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.783913 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.785470 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.785921 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.786056 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:23.786572 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:10:23.787088 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:10:23.787248 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:23.787325 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 01:10:23.787414 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 01:10:23.796228 19230 raft_consensus.cc:1261] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e [term 1 FOLLOWER]: Refusing update from remote peer 53df5158a06b4d21bcf969250316a0e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:10:23.797225 19097 raft_consensus.cc:1261] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c [term 1 FOLLOWER]: Refusing update from remote peer 53df5158a06b4d21bcf969250316a0e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:10:23.797230 19315 consensus_queue.cc:1035] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7d3b913306a4c0fb3c90e6482d1f84e" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:10:23.798055 19315 consensus_queue.cc:1035] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5aef212b824417b594b4beba5ee10c" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 41661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:10:23.812112 19318 mvcc.cc:205] Tried to move back new op lower bound from 7334478740659294208 to 7334478740435787776. Current Snapshot: MvccSnapshot[applied={T|T < 7334478740659294208}]
I20260929 01:10:23.829396 19320 mvcc.cc:205] Tried to move back new op lower bound from 7334478740659294208 to 7334478740435787776. Current Snapshot: MvccSnapshot[applied={T|T < 7334478740659294208 or (T in {7334478740659294208})}]
I20260929 01:10:23.829767 19319 mvcc.cc:205] Tried to move back new op lower bound from 7334478740659294208 to 7334478740435787776. Current Snapshot: MvccSnapshot[applied={T|T < 7334478740659294208 or (T in {7334478740659294208})}]
I20260929 01:10:23.836961 18800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:10:23.837246 18800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:10:23.841326 18800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee1a8cc38804409e89fa5ac821673678: Sending DeleteTablet for 3 replicas of tablet 2c18507df4074ab98c9513f35941d7b1
I20260929 01:10:23.842595 19077 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c18507df4074ab98c9513f35941d7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:10:23 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:34260
I20260929 01:10:23.843207 19210 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c18507df4074ab98c9513f35941d7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:10:23 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:56554
I20260929 01:10:23.843405 18932 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c18507df4074ab98c9513f35941d7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:10:23 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:48052
I20260929 01:10:23.853519 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 18837
I20260929 01:10:23.880128 19329 tablet_replica.cc:330] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9: stopping tablet replica
I20260929 01:10:23.880929 19329 raft_consensus.cc:2229] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:10:23.881904 19329 raft_consensus.cc:2258] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:10:23.882642 19327 tablet_replica.cc:330] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c: stopping tablet replica
I20260929 01:10:23.883491 19327 raft_consensus.cc:2229] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:10:23.884505 19328 tablet_replica.cc:330] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e: stopping tablet replica
I20260929 01:10:23.884984 19328 raft_consensus.cc:2229] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:10:23.885442 19328 raft_consensus.cc:2258] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:10:23.886559 19329 ts_tablet_manager.cc:1905] T 2c18507df4074ab98c9513f35941d7b1 P 53df5158a06b4d21bcf969250316a0e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:10:23.886727 19327 raft_consensus.cc:2258] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:10:23.887403 19328 ts_tablet_manager.cc:1905] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:10:23.889117 19327 ts_tablet_manager.cc:1905] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:10:23.894335 19328 ts_tablet_manager.cc:1918] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:10:23.894496 19328 log.cc:1196] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/2c18507df4074ab98c9513f35941d7b1
I20260929 01:10:23.895001 19328 ts_tablet_manager.cc:1939] T 2c18507df4074ab98c9513f35941d7b1 P c7d3b913306a4c0fb3c90e6482d1f84e: Deleting consensus metadata
I20260929 01:10:23.895061 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:23.862201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48064 (local address 127.5.198.193:36229)
0929 01:10:23.862451 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:23.862460 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:23.862488 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:10:23.862816 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:23.862825 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:23.862938 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:23.863051 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:23.867614 (+ 4563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.869013 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.871522 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:23.872485 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:23.872658 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:23.877359 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:23.877385 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:23.877412 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:23.877478 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:23.886488 (+ 9010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:23.887655 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:23.887666 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:23.887670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:23.887869 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:23.891659 (+ 3790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:23.891666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:23.891670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:23.892594 (+ 924us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:23.892906 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:23.894380 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0929 01:10:23.894637 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 01:10:23.895830 18992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:23.896847 18787 catalog_manager.cc:4841] TS c7d3b913306a4c0fb3c90e6482d1f84e (127.5.198.195:37763): tablet 2c18507df4074ab98c9513f35941d7b1 (table test-table [id=e12da2e492ac4ae0a0270ac07affce86]) successfully deleted
I20260929 01:10:23.903709 19327 ts_tablet_manager.cc:1918] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:10:23.903919 19327 log.cc:1196] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/2c18507df4074ab98c9513f35941d7b1
I20260929 01:10:23.904541 19327 ts_tablet_manager.cc:1939] T 2c18507df4074ab98c9513f35941d7b1 P bf5aef212b824417b594b4beba5ee10c: Deleting consensus metadata
I20260929 01:10:23.906390 18785 catalog_manager.cc:4841] TS bf5aef212b824417b594b4beba5ee10c (127.5.198.194:41661): tablet 2c18507df4074ab98c9513f35941d7b1 (table test-table [id=e12da2e492ac4ae0a0270ac07affce86]) successfully deleted
I20260929 01:10:24.018081 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 18837
W20260929 01:10:24.038444 18786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.198.193:36229 (error 108)
I20260929 01:10:24.038640 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 19001
I20260929 01:10:24.040352 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:24.039799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:36229 (local address 127.0.0.1:48080)
0929 01:10:24.040047 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:24.040180 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260929 01:10:24.040630 18786 catalog_manager.cc:4584] TS 53df5158a06b4d21bcf969250316a0e9 (127.5.198.193:36229): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c18507df4074ab98c9513f35941d7b1: Network error: Client connection negotiation failed: client connection to 127.5.198.193:36229: connect: Connection refused (error 111)
I20260929 01:10:24.058382 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:24.039384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.5.198.194:41661)
0929 01:10:24.039636 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:24.039642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:24.039719 (+ 77us) server_negotiation.cc:407] Connection header received
0929 01:10:24.040606 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:24.040617 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:24.040712 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:24.040895 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:24.042537 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:24.043682 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:24.045117 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:24.045510 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:24.045649 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:24.052324 (+ 6675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:24.052344 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:24.052366 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:24.052405 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:24.055507 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:24.056258 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:24.056265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:24.056268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:24.056397 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:24.056799 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:24.056804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:24.056808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:24.057175 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:24.057461 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:24.057783 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 01:10:24.058035 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 01:10:24.059073 19137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:24.094241 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:24.093715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:36229 (local address 127.0.0.1:48090)
0929 01:10:24.093952 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:24.094070 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 01:10:24.095361 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:24.094948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:36229 (local address 127.0.0.1:48100)
0929 01:10:24.095100 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:24.095198 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 01:10:24.149220 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 19001
I20260929 01:10:24.156003 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:24.155524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:36229 (local address 127.0.0.1:48110)
0929 01:10:24.155735 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:24.155835 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 01:10:24.157248 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:24.156746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:36229 (local address 127.0.0.1:48124)
0929 01:10:24.156944 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:24.157047 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 01:10:24.167611 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 19146
I20260929 01:10:24.181978 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:24.168411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56570 (local address 127.5.198.195:37763)
0929 01:10:24.168587 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:24.168591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:24.168630 (+ 39us) server_negotiation.cc:407] Connection header received
0929 01:10:24.168854 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:24.168859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:24.168942 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:24.169032 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:24.170650 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:24.171392 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:24.173018 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:24.173418 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:24.173517 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:24.176036 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:24.176054 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:24.176079 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:24.176117 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:24.178939 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:24.179624 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:24.179631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:24.179634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:24.179786 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:24.180329 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:24.180334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:24.180336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:24.180821 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:24.181086 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:24.181380 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 01:10:24.181564 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:10:24.182627 19270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:24.237663 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:24.237135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:36229 (local address 127.0.0.1:48140)
0929 01:10:24.237372 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:24.237509 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 01:10:24.238922 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:24.238446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:36229 (local address 127.0.0.1:48144)
0929 01:10:24.238664 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:24.238759 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 01:10:24.274544 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 19146
I20260929 01:10:24.291625 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 18767
I20260929 01:10:24.293071 18830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:24.424333 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:24.423413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:36229 (local address 127.0.0.1:48146)
0929 01:10:24.423949 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:24.424072 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260929 01:10:24.424613 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 18767
I20260929 01:10:24.448993 5915 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 01:10:24.784750 5915 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 01:10:24.784974 5915 mini_ranger.cc:66] Stopping Ranger...
I20260929 01:10:25.130716 5915 mini_ranger.cc:68] Stopped Ranger
2026-09-29 01:10:25.131 UTC [17819] LOG: received smart shutdown request
2026-09-29 01:10:25.134 UTC [17819] LOG: background worker "logical replication launcher" (PID 17828) exited with exit code 1
2026-09-29 01:10:25.134 UTC [17823] LOG: shutting down
2026-09-29 01:10:25.341 UTC [17819] LOG: database system is shut down
2026-09-29T01:10:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52804 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:10:25 dist-test-slave-41cl krb5kdc[19343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:10:25 dist-test-slave-41cl krb5kdc[19343](info): set up 2 sockets
Sep 29 01:10:25 dist-test-slave-41cl krb5kdc[19343](info): commencing operation
krb5kdc: starting...
W20260929 01:10:26.589378 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:10:26 dist-test-slave-41cl krb5kdc[19343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:10:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:10:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:10:26.829077 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:40133
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:40133,127.5.198.253:45751,127.5.198.252:44359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:10:27.157820 19356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:27.158111 19356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:27.158283 19356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:27.165663 19356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:10:27.165766 19356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:27.165853 19356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:27.165956 19356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:10:27.166035 19356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:10:27.175966 19356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:40133,127.5.198.253:45751,127.5.198.252:44359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:40133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:27.177356 19356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:27.178646 19356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:27.191066 19361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:27.191179 19362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:27.191211 19364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:27.191677 19356 server_base.cc:1034] running on GCE node
I20260929 01:10:27.192730 19356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:27.194088 19356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:27.195324 19356 hybrid_clock.cc:648] HybridClock initialized: now 1790644227195273 us; error 42 us; skew 500 ppm
I20260929 01:10:27.195693 19356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:27 dist-test-slave-41cl krb5kdc[19343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644227, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:27.198844 19356 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:10:27.200345 19356 webserver.cc:460] Webserver started at http://127.5.198.254:39949/ using document root <none> and password file <none>
I20260929 01:10:27.201054 19356 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:27.201152 19356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:27.201385 19356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:10:27.204226 19356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "5488f70e07754a57ba2be20f6e75b96e"
format_stamp: "Formatted at 2026-09-29 01:10:27 on dist-test-slave-41cl"
server_key: "c555c8770ee0cd733e33869f1bd09a83"
server_key_iv: "09c27c6a699ee89ff8de035f234dd5b6"
server_key_version: "encryptionkey@0"
I20260929 01:10:27.205071 19356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "5488f70e07754a57ba2be20f6e75b96e"
format_stamp: "Formatted at 2026-09-29 01:10:27 on dist-test-slave-41cl"
server_key: "c555c8770ee0cd733e33869f1bd09a83"
server_key_iv: "09c27c6a699ee89ff8de035f234dd5b6"
server_key_version: "encryptionkey@0"
I20260929 01:10:27.210922 19356 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:10:27.214751 19371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:27.216233 19356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:10:27.216449 19356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "5488f70e07754a57ba2be20f6e75b96e"
format_stamp: "Formatted at 2026-09-29 01:10:27 on dist-test-slave-41cl"
server_key: "c555c8770ee0cd733e33869f1bd09a83"
server_key_iv: "09c27c6a699ee89ff8de035f234dd5b6"
server_key_version: "encryptionkey@0"
I20260929 01:10:27.216709 19356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:27.245424 19356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:27.249440 19356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:27.249733 19356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:27.270560 19356 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:40133
I20260929 01:10:27.270579 19422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:40133 every 8 connection(s)
I20260929 01:10:27.271991 19356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:10:27.275688 19423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:10:27.281772 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 19356
I20260929 01:10:27.282001 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:10:27.282788 5915 external_mini_cluster.cc:1428] Setting key ef7fe25d24cae7591419acb531fab0a9
I20260929 01:10:27.282344 19423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:10:27 dist-test-slave-41cl krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644227, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.198.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.253@KRBTEST.COM" created.
I20260929 01:10:27.307139 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.286114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:40133 (local address 127.0.0.1:35814)
0929 01:10:27.287041 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.287098 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:27.290040 (+ 2942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:27.290948 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:27.290955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:27.291690 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:27.292029 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:27.292046 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:27.293641 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:27.293647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:27.294711 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:27.294722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:27.295619 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:27.295626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:27.295799 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:27.296626 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:27.296651 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:27.299246 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:27.302763 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:27.302776 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:27.302788 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:27.303544 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:27.303986 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:27.303991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:27.303995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:27.304205 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:27.305078 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:27.305114 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:27.305326 (+ 212us) client_negotiation.cc:769] Sending connection context
0929 01:10:27.305557 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 01:10:27.306149 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":185,"thread_start_us":388,"threads_started":1}
I20260929 01:10:27.307190 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.286114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.5.198.254:40133)
0929 01:10:27.287996 (+ 1882us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:27.288002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:27.288043 (+ 41us) server_negotiation.cc:407] Connection header received
0929 01:10:27.290460 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:27.290465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:27.290704 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:27.290947 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:27.292224 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:27.293487 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:27.294855 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:27.295392 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:27.295591 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:27.299520 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:27.299529 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:27.299542 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:27.299571 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:27.302613 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:27.303704 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:27.303711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:27.303713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:27.303856 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:27.304332 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:27.304337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:27.304342 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:27.304911 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:27.305157 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:27.305691 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 01:10:27.305952 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1700,"spinlock_wait_cycles":22272,"thread_start_us":1465,"threads_started":1}
I20260929 01:10:27.310245 19423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:10:27.311564 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.311054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:45751 (local address 127.0.0.1:45496)
0929 01:10:27.311267 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.311403 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 01:10:27.311864 19375 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
I20260929 01:10:27.313593 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.313147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:45751 (local address 127.0.0.1:45506)
0929 01:10:27.313349 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.313444 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 01:10:27.313787 19423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.253:45751: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.253.keytab.
Entry for principal kudu/127.5.198.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.253.keytab.
I20260929 01:10:27.337198 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.253:45751
--webserver_interface=127.5.198.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.253.keytab
--principal=kudu/127.5.198.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:40133,127.5.198.253:45751,127.5.198.252:44359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:10:27.363147 19423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } attempt: 1
I20260929 01:10:27.364665 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.364061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:45751 (local address 127.0.0.1:45508)
0929 01:10:27.364324 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.364487 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 01:10:27.365760 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.365356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:45751 (local address 127.0.0.1:45512)
0929 01:10:27.365515 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.365605 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 01:10:27.366045 19423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.253:45751: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
I20260929 01:10:27.434298 19423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } attempt: 2
I20260929 01:10:27.435621 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.435108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:45751 (local address 127.0.0.1:45518)
0929 01:10:27.435367 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.435483 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 01:10:27.436694 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.436278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:45751 (local address 127.0.0.1:45530)
0929 01:10:27.436431 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.436573 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 01:10:27.436879 19423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.253:45751: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
I20260929 01:10:27.528144 19423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } attempt: 3
I20260929 01:10:27.529444 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.528973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:45751 (local address 127.0.0.1:45542)
0929 01:10:27.529197 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.529304 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 01:10:27.530519 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.530152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:45751 (local address 127.0.0.1:45558)
0929 01:10:27.530314 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.530384 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 01:10:27.530704 19423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.253:45751: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
W20260929 01:10:27.668349 19430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:27.668717 19430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:27.668946 19430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 01:10:27.673970 19423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } attempt: 4
I20260929 01:10:27.675186 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.674604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:45751 (local address 127.0.0.1:45564)
0929 01:10:27.674968 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.675053 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260929 01:10:27.676139 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.675801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:45751 (local address 127.0.0.1:45578)
0929 01:10:27.675943 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.676031 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 01:10:27.676321 19423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.253:45751: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
W20260929 01:10:27.676856 19430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:10:27.677001 19430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:27.677109 19430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:27.677219 19430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:10:27.677310 19430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:10:27.687251 19430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:40133,127.5.198.253:45751,127.5.198.252:44359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.253.keytab
--principal=kudu/127.5.198.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.253:45751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.198.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:27.688720 19430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:27.690092 19430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:27.702000 19435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:27.701968 19438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:27.702033 19436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:27.702811 19430 server_base.cc:1034] running on GCE node
I20260929 01:10:27.703531 19430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:27.704980 19430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:27.706226 19430 hybrid_clock.cc:648] HybridClock initialized: now 1790644227706187 us; error 47 us; skew 500 ppm
I20260929 01:10:27.706595 19430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:27 dist-test-slave-41cl krb5kdc[19343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644227, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:27.709975 19430 init.cc:377] Logged in from keytab as kudu/127.5.198.253@KRBTEST.COM (short username kudu)
I20260929 01:10:27.711563 19430 webserver.cc:460] Webserver started at http://127.5.198.253:46023/ using document root <none> and password file <none>
I20260929 01:10:27.712268 19430 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:27.712432 19430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:27.712772 19430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:10:27.716006 19430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/data/instance:
uuid: "a8adcdca8e32489f8d243faa7ca199a0"
format_stamp: "Formatted at 2026-09-29 01:10:27 on dist-test-slave-41cl"
server_key: "83aef1b5d2a8ecdcac37cafa56ca3a1a"
server_key_iv: "004d134a717a10516b5ad6da1c87b666"
server_key_version: "encryptionkey@0"
I20260929 01:10:27.717067 19430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/wal/instance:
uuid: "a8adcdca8e32489f8d243faa7ca199a0"
format_stamp: "Formatted at 2026-09-29 01:10:27 on dist-test-slave-41cl"
server_key: "83aef1b5d2a8ecdcac37cafa56ca3a1a"
server_key_iv: "004d134a717a10516b5ad6da1c87b666"
server_key_version: "encryptionkey@0"
I20260929 01:10:27.722561 19430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 01:10:27.726025 19445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:27.727385 19430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:10:27.727582 19430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/wal
uuid: "a8adcdca8e32489f8d243faa7ca199a0"
format_stamp: "Formatted at 2026-09-29 01:10:27 on dist-test-slave-41cl"
server_key: "83aef1b5d2a8ecdcac37cafa56ca3a1a"
server_key_iv: "004d134a717a10516b5ad6da1c87b666"
server_key_version: "encryptionkey@0"
I20260929 01:10:27.727784 19430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:27.740456 19430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:27.744868 19430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:27.745150 19430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:27.764425 19430 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.253:45751
I20260929 01:10:27.764544 19496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.253:45751 every 8 connection(s)
I20260929 01:10:27.765823 19430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/data/info.pb
I20260929 01:10:27.769493 19497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:10:27.769763 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 19430
I20260929 01:10:27.769955 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-1/wal/instance
I20260929 01:10:27.770399 5915 external_mini_cluster.cc:1428] Setting key a984db9ff882c6f6861de0d07ce01030
I20260929 01:10:27.776134 19497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.198.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.252@KRBTEST.COM" created.
Sep 29 01:10:27 dist-test-slave-41cl krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644227, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.253@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:27.800372 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.779304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35828 (local address 127.5.198.254:40133)
0929 01:10:27.779522 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:27.779529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:27.780799 (+ 1270us) server_negotiation.cc:407] Connection header received
0929 01:10:27.782622 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:27.782630 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:27.782723 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:27.782854 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:27.784607 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:27.785836 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:27.787593 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:27.788411 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:27.788592 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:27.794595 (+ 6003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:27.794608 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:27.794611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:27.794644 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:27.797350 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:27.798078 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:27.798089 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:27.798092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:27.798202 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:27.798631 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:27.798636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:27.798639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:27.798921 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:27.799098 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:27.799819 (+ 721us) server_negotiation.cc:299] Negotiation successful
0929 01:10:27.800054 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 01:10:27.801409 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.779760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:40133 (local address 127.0.0.1:35828)
0929 01:10:27.780626 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.780678 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:27.782360 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:27.783164 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:27.783174 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:27.783844 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:27.784411 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:27.784427 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:27.786008 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:27.786014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:27.787437 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:27.787449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:27.788653 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:27.788658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:27.788849 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:27.789779 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:27.789832 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:27.794361 (+ 4529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:27.797492 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:27.797501 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:27.797525 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:27.797939 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:27.798324 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:27.798329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:27.798332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:27.798505 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:27.799066 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:27.799074 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:27.799451 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 01:10:27.799719 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 01:10:27.800214 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":356,"threads_started":1}
I20260929 01:10:27.803153 19497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:10:27 dist-test-slave-41cl krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644227, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.253@KRBTEST.COM for kudu/127.5.198.253@KRBTEST.COM
I20260929 01:10:27.818665 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.803946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:45751 (local address 127.0.0.1:45592)
0929 01:10:27.804147 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.804168 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:27.804301 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:27.805203 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:27.805207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:27.805571 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:27.805702 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:27.805710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:27.807750 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:27.807754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:27.808727 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:27.808744 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:27.810018 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:27.810022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:27.810245 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:27.810868 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:27.810928 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:27.812880 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:27.816095 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:27.816100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:27.816102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:27.816457 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:27.817023 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:27.817026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:27.817028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:27.817173 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:27.817885 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:27.817888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:27.817995 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 01:10:27.818109 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 01:10:27.818377 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":24}
I20260929 01:10:27.818971 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.803977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.5.198.253:45751)
0929 01:10:27.804698 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:27.804708 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:27.804741 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:10:27.804796 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:27.804806 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:27.805039 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:27.805188 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:27.805859 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:27.807596 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:27.808889 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:27.809846 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:27.809998 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:27.813104 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:27.813123 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:27.813136 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:27.813171 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:27.815927 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:27.816720 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:27.816732 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:27.816739 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:27.816920 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:27.817387 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:27.817391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:27.817394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:27.817766 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:27.817915 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:27.818342 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 01:10:27.818564 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":507,"thread_start_us":269,"threads_started":1}
I20260929 01:10:27.821571 19497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:10:27.822662 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.822223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56760)
0929 01:10:27.822412 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.822509 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 01:10:27.822892 19446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.252.keytab.
Entry for principal kudu/127.5.198.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.252.keytab.
I20260929 01:10:27.824716 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.824244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56770)
0929 01:10:27.824403 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.824569 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 01:10:27.824942 19497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:44359: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
I20260929 01:10:27.824970 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.252:44359
--webserver_interface=127.5.198.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.252.keytab
--principal=kudu/127.5.198.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:40133,127.5.198.253:45751,127.5.198.252:44359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:10:27.874279 19497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } attempt: 1
I20260929 01:10:27.875515 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.875040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56782)
0929 01:10:27.875243 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.875357 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 01:10:27.876579 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.876178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56786)
0929 01:10:27.876346 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.876412 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 01:10:27.876778 19497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:44359: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
I20260929 01:10:27.945011 19497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } attempt: 2
I20260929 01:10:27.946180 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.945720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56794)
0929 01:10:27.945921 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.946028 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 01:10:27.947233 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.946897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56800)
0929 01:10:27.947056 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.947118 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 01:10:27.947434 19497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:44359: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
I20260929 01:10:27.975584 19423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } attempt: 5
Sep 29 01:10:27 dist-test-slave-41cl krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644227, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for kudu/127.5.198.253@KRBTEST.COM
I20260929 01:10:27.990896 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.976293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:45751 (local address 127.0.0.1:45602)
0929 01:10:27.976574 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.976599 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:27.976730 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:27.977098 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:27.977103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:27.977589 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:27.977815 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:27.977828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:27.978848 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:27.978853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:27.980228 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:27.980244 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:27.981027 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:27.981035 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:27.981221 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:27.981873 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:27.981913 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:27.984586 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:27.988382 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:27.988386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:27.988389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:27.988813 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:27.989245 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:27.989248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:27.989251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:27.989372 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:27.990137 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:27.990141 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:27.990280 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 01:10:27.990431 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 01:10:27.990644 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 01:10:27.991186 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.976451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45602 (local address 127.5.198.253:45751)
0929 01:10:27.976688 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:27.976692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:27.976711 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:10:27.976855 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:27.976859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:27.976925 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:27.977074 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:27.978019 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:27.978707 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:27.980376 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:27.980861 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:27.980965 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:27.984775 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:27.984782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:27.984787 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:27.984825 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:27.988237 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:27.988937 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:27.988944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:27.988949 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:27.989121 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:27.989503 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:27.989509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:27.989515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:27.989971 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:27.990165 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:27.990625 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 01:10:27.990872 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 01:10:27.992250 19423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:10:27.993289 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.992908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56810)
0929 01:10:27.993096 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.993169 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 01:10:27.994186 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:27.993869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56820)
0929 01:10:27.994031 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:27.994097 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 01:10:27.994377 19423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:44359: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
I20260929 01:10:28.038653 19497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } attempt: 3
I20260929 01:10:28.039800 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.039334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56826)
0929 01:10:28.039543 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:28.039644 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 01:10:28.040874 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.040430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56834)
0929 01:10:28.040697 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:28.040767 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
W20260929 01:10:28.041096 19497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:44359: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
I20260929 01:10:28.045601 19423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } attempt: 1
I20260929 01:10:28.046627 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.046202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56838)
0929 01:10:28.046385 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:28.046481 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 01:10:28.047545 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.047221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56848)
0929 01:10:28.047392 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:28.047451 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 01:10:28.047744 19423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:44359: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
I20260929 01:10:28.115965 19423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } attempt: 2
I20260929 01:10:28.117235 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.116712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56860)
0929 01:10:28.116986 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:28.117074 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 01:10:28.118227 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.117912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56870)
0929 01:10:28.118057 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:28.118127 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 01:10:28.118414 19423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:44359: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
W20260929 01:10:28.167753 19504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:28.168087 19504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:28.168253 19504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:28.175437 19504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:10:28.175580 19504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:28.175688 19504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:28.175791 19504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:10:28.175861 19504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:10:28.184352 19497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } attempt: 4
I20260929 01:10:28.185658 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.185100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56880)
0929 01:10:28.185360 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:28.185467 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 01:10:28.186141 19504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:40133,127.5.198.253:45751,127.5.198.252:44359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.252.keytab
--principal=kudu/127.5.198.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.252:44359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.198.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:28.186707 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.186318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56886)
0929 01:10:28.186523 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:28.186586 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260929 01:10:28.186952 19497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:44359: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
I20260929 01:10:28.187676 19504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:28.189044 19504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:28.201189 19510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:28.201292 19509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:28.201265 19512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:28.202360 19504 server_base.cc:1034] running on GCE node
I20260929 01:10:28.203174 19504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:28.204608 19504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:28.205854 19504 hybrid_clock.cc:648] HybridClock initialized: now 1790644228205786 us; error 61 us; skew 500 ppm
I20260929 01:10:28.206315 19504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:28 dist-test-slave-41cl krb5kdc[19343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644228, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:28.209542 19504 init.cc:377] Logged in from keytab as kudu/127.5.198.252@KRBTEST.COM (short username kudu)
I20260929 01:10:28.211032 19504 webserver.cc:460] Webserver started at http://127.5.198.252:39497/ using document root <none> and password file <none>
I20260929 01:10:28.211699 19504 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:28.211838 19504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:28.212139 19504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:10:28.215154 19504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/data/instance:
uuid: "b1ababa31bc64e35b2658974081d0c38"
format_stamp: "Formatted at 2026-09-29 01:10:28 on dist-test-slave-41cl"
server_key: "0894332a41f89c90a63e7a66d58a17cc"
server_key_iv: "0a9bf679be85c0f59e004211a46415b0"
server_key_version: "encryptionkey@0"
I20260929 01:10:28.216012 19504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/wal/instance:
uuid: "b1ababa31bc64e35b2658974081d0c38"
format_stamp: "Formatted at 2026-09-29 01:10:28 on dist-test-slave-41cl"
server_key: "0894332a41f89c90a63e7a66d58a17cc"
server_key_iv: "0a9bf679be85c0f59e004211a46415b0"
server_key_version: "encryptionkey@0"
I20260929 01:10:28.221292 19504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:10:28.224627 19423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } attempt: 3
I20260929 01:10:28.224854 19519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:28.225833 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.225264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56890)
0929 01:10:28.225535 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:28.225634 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260929 01:10:28.226588 19504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:10:28.226889 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.226510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56898)
0929 01:10:28.226715 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:28.226786 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 01:10:28.226787 19504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/wal
uuid: "b1ababa31bc64e35b2658974081d0c38"
format_stamp: "Formatted at 2026-09-29 01:10:28 on dist-test-slave-41cl"
server_key: "0894332a41f89c90a63e7a66d58a17cc"
server_key_iv: "0a9bf679be85c0f59e004211a46415b0"
server_key_version: "encryptionkey@0"
W20260929 01:10:28.227084 19423 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:44359: Network error: Client connection negotiation failed: client connection to 127.5.198.252:44359: connect: Connection refused (error 111)
I20260929 01:10:28.227015 19504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:28.244694 19504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:28.248999 19504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:28.249281 19504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:28.271370 19504 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.252:44359
I20260929 01:10:28.271382 19570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.252:44359 every 8 connection(s)
I20260929 01:10:28.273262 19504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/data/info.pb
I20260929 01:10:28.276201 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 19504
I20260929 01:10:28.276393 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790644023292677-5915-0/minicluster-data/master-2/wal/instance
I20260929 01:10:28.277191 5915 external_mini_cluster.cc:1428] Setting key 22be19006bd2b6ba8c14504cffa03de6
I20260929 01:10:28.279208 19571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:10:28.290820 19571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:10:28 dist-test-slave-41cl krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.252@KRBTEST.COM
Sep 29 01:10:28 dist-test-slave-41cl krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644228, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.252@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:28.306984 19574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.279412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.5.198.252:44359)
0929 01:10:28.285120 (+ 5708us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:28.285142 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:28.285768 (+ 626us) server_negotiation.cc:407] Connection header received
0929 01:10:28.287938 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:28.287962 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:28.288369 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:28.288884 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:28.290355 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.291679 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.293391 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.294553 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.294743 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:28.299292 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:28.299299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:28.299314 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:28.299351 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:28.302602 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.303440 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.303452 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.303459 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.303600 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.303993 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.304004 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.304006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.304695 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:28.305087 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:28.305538 (+ 451us) server_negotiation.cc:299] Negotiation successful
0929 01:10:28.305972 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5393,"thread_start_us":5176,"threads_started":1}
I20260929 01:10:28.308172 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.293549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.5.198.254:40133)
0929 01:10:28.293742 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:28.293746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:28.294453 (+ 707us) server_negotiation.cc:407] Connection header received
0929 01:10:28.294707 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:28.294713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:28.294789 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:28.294903 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:28.296047 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.296867 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.298609 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.299077 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.299234 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:28.302637 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:28.302642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:28.302646 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:28.302678 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:28.305399 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.306225 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.306232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.306237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.306368 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.306728 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.306734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.306737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.306976 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:28.307141 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:28.307583 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 01:10:28.307811 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 01:10:28.308212 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.293465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:40133 (local address 127.0.0.1:35834)
0929 01:10:28.294304 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:28.294341 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:28.294546 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:28.294971 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:28.294978 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:28.295584 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:28.295871 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:28.295880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.297030 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.297036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:28.298457 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:28.298470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.299344 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.299352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:28.299530 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:28.300172 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:28.300198 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:28.302422 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:28.305531 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:28.305544 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:28.305551 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:28.306114 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:28.306477 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:28.306482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:28.306486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:28.306616 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:28.307115 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:28.307122 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:28.307292 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 01:10:28.307466 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 01:10:28.307799 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":449,"threads_started":1}
I20260929 01:10:28.309764 19571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:10:28 dist-test-slave-41cl krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644228, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.252@KRBTEST.COM for kudu/127.5.198.253@KRBTEST.COM
Sep 29 01:10:28 dist-test-slave-41cl krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:28.323570 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.310536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:45751 (local address 127.0.0.1:45616)
0929 01:10:28.310757 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:28.310774 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:28.310877 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:28.311246 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:28.311249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:28.311600 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:28.311721 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:28.311728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.312765 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.312771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:28.313639 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:28.313650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.314329 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.314335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:28.314538 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:28.315075 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:28.315090 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:28.317680 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:28.321082 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:28.321089 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:28.321096 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:28.321514 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:28.322011 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:28.322017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:28.322022 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:28.322219 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:28.322859 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:28.322862 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:28.323029 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 01:10:28.323153 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 01:10:28.323343 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260929 01:10:28.323920 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.310802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45616 (local address 127.5.198.253:45751)
0929 01:10:28.311002 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:28.311006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:28.311020 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:10:28.311067 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:28.311070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:28.311124 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:28.311228 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:28.311866 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.312635 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.313776 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.314175 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.314312 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:28.317869 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:28.317875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:28.317878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:28.317907 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:28.320903 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.321649 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.321655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.321659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.321810 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.322361 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.322368 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.322372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.322734 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:28.322903 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:28.323296 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 01:10:28.323550 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:10:28.325114 19571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:10:28.326800 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.310651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.5.198.254:40133)
0929 01:10:28.310839 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:28.310844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:28.310863 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:10:28.310996 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:28.311002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:28.311083 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:28.311191 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:28.312550 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.313342 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.314722 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.315172 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.315311 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:28.320273 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:28.320279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:28.320286 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:28.320325 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:28.323820 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.324737 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.324746 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.324752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.324912 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.325295 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.325299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.325302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.325645 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:28.325991 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:28.326283 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 01:10:28.326519 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 01:10:28 dist-test-slave-41cl krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644228, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.252@KRBTEST.COM for kudu/127.5.198.252@KRBTEST.COM
Sep 29 01:10:28 dist-test-slave-41cl krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.253@KRBTEST.COM
I20260929 01:10:28.340595 19574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.325863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.5.198.252:44359)
0929 01:10:28.326023 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:28.326030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:28.326204 (+ 174us) server_negotiation.cc:407] Connection header received
0929 01:10:28.326340 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:28.326345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:28.326430 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:28.326521 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:28.327317 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.328121 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.329675 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.330381 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.330491 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:28.334270 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:28.334276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:28.334281 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:28.334318 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:28.337716 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.338513 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.338519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.338523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.338658 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.339107 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.339112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.339116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.339448 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:28.339596 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:28.339930 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 01:10:28.340186 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":58}
I20260929 01:10:28.340673 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.325815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56914)
0929 01:10:28.326118 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:28.326137 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:28.326232 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:28.326573 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:28.326579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:28.327045 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:28.327191 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:28.327199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.328254 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.328467 (+ 213us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:28.329536 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:28.329547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.330530 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.330536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:28.330726 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:28.331338 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:28.331384 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:28.334074 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:28.337902 (+ 3828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:28.337910 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:28.337914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:28.338385 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:28.338803 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:28.338807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:28.338809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:28.338986 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:28.339561 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:28.339566 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:28.339696 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 01:10:28.339849 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 01:10:28.340330 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":155}
I20260929 01:10:28.344861 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.328374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.5.198.253:45751)
0929 01:10:28.328552 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:28.328558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:28.328577 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:10:28.328695 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:28.328699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:28.328767 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:28.328911 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:28.330181 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.331278 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.333122 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.333825 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.333965 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:28.339358 (+ 5393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:28.339363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:28.339366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:28.339394 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:28.342279 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.342950 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.342956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.342959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.343103 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.343483 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.343488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.343491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.343740 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:28.344027 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:28.344285 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 01:10:28.344567 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:10:28.345676 19571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38: Bootstrap starting.
I20260929 01:10:28.349922 19571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38: Neither blocks nor log segments found. Creating new log.
I20260929 01:10:28.350131 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:10:28.351105 19571 log.cc:824] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:28.354236 19571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38: No bootstrap required, opened a new log
I20260929 01:10:28.368908 19571 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } }
I20260929 01:10:28.369604 19571 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:10:28.369804 19571 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ababa31bc64e35b2658974081d0c38, State: Initialized, Role: FOLLOWER
I20260929 01:10:28.370767 19571 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } }
I20260929 01:10:28.374020 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.354889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35850 (local address 127.5.198.254:40133)
0929 01:10:28.355084 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:28.355092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:28.355867 (+ 775us) server_negotiation.cc:407] Connection header received
0929 01:10:28.356058 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:28.356064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:28.356136 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:28.356239 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:28.358709 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.359850 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.361626 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.362282 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.362415 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:28.366366 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:28.366374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:28.366380 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:28.366427 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:28.370608 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.371445 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.371451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.371454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.371597 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.372096 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.372103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.372110 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.372595 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:28.372853 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:28.373411 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 01:10:28.373671 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:10:28.375501 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.354889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45626 (local address 127.5.198.253:45751)
0929 01:10:28.355534 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:28.355541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:28.357088 (+ 1547us) server_negotiation.cc:407] Connection header received
0929 01:10:28.357364 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:28.357373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:28.357460 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:28.357533 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:28.359762 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.360900 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.363613 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.364355 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.364457 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:28.368403 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:28.368412 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:28.368422 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:28.368498 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:28.372064 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.372993 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.372998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.373001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.373158 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.373581 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.373588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.373593 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.374000 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:28.374306 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:28.374896 (+ 590us) server_negotiation.cc:299] Negotiation successful
0929 01:10:28.375135 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489}
I20260929 01:10:28.378221 19584 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } } }
I20260929 01:10:28.378679 19584 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:10:28.380738 19571 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:10:28.388288 19574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.357588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56926 (local address 127.5.198.252:44359)
0929 01:10:28.357775 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:28.357781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:28.367514 (+ 9733us) server_negotiation.cc:407] Connection header received
0929 01:10:28.368008 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:28.368016 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:28.368099 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:28.368174 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:28.370081 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.371192 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.374110 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.375126 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.375273 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:28.380123 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:28.380129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:28.380132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:28.380164 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:28.384407 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.385501 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.385508 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.385511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.385748 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.386320 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.386328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.386334 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.386787 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:28.387426 (+ 639us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:28.387591 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 01:10:28.387830 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 01:10:28.391501 19595 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:10:28.391691 19595 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:10:28.388883 19504 master_runner.cc:186] Error getting master registration for 127.5.198.254:40133: OK, instance_id { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" instance_seqno: 1790644227264302 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:10:28.399778 19504 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:10:28.404306 19423 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } attempt: 4
Sep 29 01:10:28 dist-test-slave-41cl krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644227, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for kudu/127.5.198.252@KRBTEST.COM
I20260929 01:10:28.420135 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.404977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56928)
0929 01:10:28.405171 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:28.405191 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:28.405292 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:28.405646 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:28.405649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:28.406094 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:28.406294 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:28.406310 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.407879 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.407884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:28.408957 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:28.408969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.410019 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.410024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:28.410165 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:28.410888 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:28.410906 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:28.413302 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:28.417640 (+ 4338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:28.417645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:28.417649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:28.418035 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:28.418404 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:28.418411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:28.418415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:28.418560 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:28.419081 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:28.419086 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:28.419295 (+ 209us) client_negotiation.cc:769] Sending connection context
0929 01:10:28.419451 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 01:10:28.419742 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 01:10:28.420169 19574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.405045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.5.198.252:44359)
0929 01:10:28.405210 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:28.405216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:28.405240 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:10:28.405420 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:28.405426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:28.405507 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:28.405610 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:28.406453 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.407723 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.409091 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.409795 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.409932 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:28.413489 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:28.413496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:28.413501 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:28.413538 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:28.417505 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.418158 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.418165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.418173 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.418294 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.418669 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.418675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.418678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.418952 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:28.419114 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:28.419557 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 01:10:28.419766 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:10:28.426291 19423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e: Bootstrap starting.
I20260929 01:10:28.430647 19423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e: Neither blocks nor log segments found. Creating new log.
I20260929 01:10:28.432025 19423 log.cc:824] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:28.435559 19423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e: No bootstrap required, opened a new log
I20260929 01:10:28.446648 19423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } }
I20260929 01:10:28.447077 19423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:10:28.447204 19423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5488f70e07754a57ba2be20f6e75b96e, State: Initialized, Role: FOLLOWER
I20260929 01:10:28.447885 19423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } }
I20260929 01:10:28.449661 19598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } } }
I20260929 01:10:28.450112 19598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:10:28.451032 19423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:10:28.462878 19609 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:10:28.463093 19609 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:10:28.460235 19356 master_runner.cc:186] Error getting master registration for 127.5.198.253:45751: OK, instance_id { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" instance_seqno: 1790644227759107 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:10:28.471614 19356 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:10:28.472142 19584 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:10:28.472426 19584 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } }
I20260929 01:10:28.474064 19584 leader_election.cc:288] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5488f70e07754a57ba2be20f6e75b96e (127.5.198.254:40133), a8adcdca8e32489f8d243faa7ca199a0 (127.5.198.253:45751)
I20260929 01:10:28.474900 19398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b1ababa31bc64e35b2658974081d0c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5488f70e07754a57ba2be20f6e75b96e" is_pre_election: true
I20260929 01:10:28.475472 19398 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1ababa31bc64e35b2658974081d0c38 in term 0.
I20260929 01:10:28.476046 19520 leader_election.cc:302] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5488f70e07754a57ba2be20f6e75b96e, b1ababa31bc64e35b2658974081d0c38; no voters:
I20260929 01:10:28.476449 19584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:10:28.476641 19584 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:10:28.476742 19584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:10:28.480069 19584 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } }
I20260929 01:10:28.481212 19398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b1ababa31bc64e35b2658974081d0c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5488f70e07754a57ba2be20f6e75b96e"
I20260929 01:10:28.481426 19398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:10:28.481616 19584 leader_election.cc:288] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [CANDIDATE]: Term 1 election: Requested vote from peers 5488f70e07754a57ba2be20f6e75b96e (127.5.198.254:40133), a8adcdca8e32489f8d243faa7ca199a0 (127.5.198.253:45751)
I20260929 01:10:28.484787 19398 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1ababa31bc64e35b2658974081d0c38 in term 1.
I20260929 01:10:28.485273 19520 leader_election.cc:302] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5488f70e07754a57ba2be20f6e75b96e, b1ababa31bc64e35b2658974081d0c38; no voters:
I20260929 01:10:28.485682 19584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:10:28.486176 19497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } attempt: 5
Sep 29 01:10:28 dist-test-slave-41cl krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644227, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.253@KRBTEST.COM for kudu/127.5.198.252@KRBTEST.COM
I20260929 01:10:28.498159 19584 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [term 1 LEADER]: Becoming Leader. State: Replica: b1ababa31bc64e35b2658974081d0c38, State: Running, Role: LEADER
I20260929 01:10:28.499276 19584 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } }
W20260929 01:10:28.501358 19525 tablet.cc:2369] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:10:28.502723 19377 tablet.cc:2369] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:10:28.510496 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.486689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:44359 (local address 127.0.0.1:56944)
0929 01:10:28.486933 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:28.486953 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:28.487059 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:28.487582 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:28.487587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:28.488138 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:28.488354 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:28.488368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.489962 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.489968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:28.491397 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:28.491414 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.492741 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.492748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:28.492972 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:28.493666 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:28.493690 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:28.499459 (+ 5769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:28.505914 (+ 6455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:28.505919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:28.505924 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:28.506712 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:28.507464 (+ 752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:28.507469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:28.507473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:28.507728 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:28.508820 (+ 1092us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:28.508825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:28.509069 (+ 244us) client_negotiation.cc:769] Sending connection context
0929 01:10:28.509285 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 01:10:28.509743 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260929 01:10:28.512678 19574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:28.486769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56944 (local address 127.5.198.252:44359)
0929 01:10:28.487013 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:28.487021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:28.487045 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:10:28.487192 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:28.487198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:28.487281 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:28.487379 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:28.488560 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.489715 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.491602 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:28.492329 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:28.492458 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:28.499694 (+ 7236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:28.499700 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:28.499705 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:28.499743 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:28.505701 (+ 5958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.506978 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.506983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.506988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.507256 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:28.507980 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:28.507985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:28.507989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:28.508611 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:28.510585 (+ 1974us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:28.511545 (+ 960us) server_negotiation.cc:299] Negotiation successful
0929 01:10:28.511944 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:10:28.518667 19497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0: Bootstrap starting.
I20260929 01:10:28.525023 19497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0: Neither blocks nor log segments found. Creating new log.
I20260929 01:10:28.527691 19497 log.cc:824] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:28.527603 19611 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1ababa31bc64e35b2658974081d0c38. Latest consensus state: current_term: 1 leader_uuid: "b1ababa31bc64e35b2658974081d0c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } } }
I20260929 01:10:28.528041 19611 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:28.529336 19613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:10:28.533234 19613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:10:28.533818 19497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0: No bootstrap required, opened a new log
I20260929 01:10:28.545938 19497 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } }
I20260929 01:10:28.546458 19497 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:10:28.546614 19497 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8adcdca8e32489f8d243faa7ca199a0, State: Initialized, Role: FOLLOWER
I20260929 01:10:28.547394 19497 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } }
I20260929 01:10:28.551573 19616 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } } }
I20260929 01:10:28.552037 19616 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:10:28.553223 19497 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:10:28.553277 19472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b1ababa31bc64e35b2658974081d0c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcdca8e32489f8d243faa7ca199a0" is_pre_election: true
I20260929 01:10:28.553856 19472 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1ababa31bc64e35b2658974081d0c38 in term 0.
I20260929 01:10:28.553234 19471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b1ababa31bc64e35b2658974081d0c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8adcdca8e32489f8d243faa7ca199a0"
I20260929 01:10:28.555449 19471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:10:28.556412 19398 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e [term 1 FOLLOWER]: Refusing update from remote peer b1ababa31bc64e35b2658974081d0c38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:10:28.557551 19584 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:10:28.558735 19471 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1ababa31bc64e35b2658974081d0c38 in term 1.
I20260929 01:10:28.562780 19471 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0 [term 1 FOLLOWER]: Refusing update from remote peer b1ababa31bc64e35b2658974081d0c38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:10:28.563663 19584 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:10:28.585924 19598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1ababa31bc64e35b2658974081d0c38. Latest consensus state: current_term: 1 leader_uuid: "b1ababa31bc64e35b2658974081d0c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } } }
I20260929 01:10:28.586261 19598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:10:28.590405 19615 mvcc.cc:205] Tried to move back new op lower bound from 7334478760156164096 to 7334478759943032832. Current Snapshot: MvccSnapshot[applied={T|T < 7334478760156164096}]
I20260929 01:10:28.600227 19616 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1ababa31bc64e35b2658974081d0c38. Latest consensus state: current_term: 1 leader_uuid: "b1ababa31bc64e35b2658974081d0c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } } }
I20260929 01:10:28.600572 19616 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:10:28.606923 19616 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b1ababa31bc64e35b2658974081d0c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } } }
I20260929 01:10:28.607259 19616 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:10:28.607800 19598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b1ababa31bc64e35b2658974081d0c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } } }
I20260929 01:10:28.608151 19598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:10:28.612995 19611 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b1ababa31bc64e35b2658974081d0c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } } }
I20260929 01:10:28.613265 19611 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:28.613505 19611 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b1ababa31bc64e35b2658974081d0c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5488f70e07754a57ba2be20f6e75b96e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40133 } } peers { permanent_uuid: "a8adcdca8e32489f8d243faa7ca199a0" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 45751 } } peers { permanent_uuid: "b1ababa31bc64e35b2658974081d0c38" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 44359 } } }
I20260929 01:10:28.613699 19611 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:28.614413 19613 catalog_manager.cc:1347] Generated new cluster ID: 483357c80b9446be8a5b58e833a5645b
I20260929 01:10:28.614526 19613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:10:28.639006 19613 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:10:28.640972 19613 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:10:28.667290 19613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1ababa31bc64e35b2658974081d0c38: Generated new TSK 0
I20260929 01:10:28.668505 19613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:10:28.699270 19634 catalog_manager.cc:1259] Loaded cluster ID: 483357c80b9446be8a5b58e833a5645b
I20260929 01:10:28.699491 19634 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0: loading cluster ID for follower catalog manager: success
I20260929 01:10:28.703097 19634 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0: acquiring CA information for follower catalog manager: success
I20260929 01:10:28.709029 19634 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a8adcdca8e32489f8d243faa7ca199a0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:10:29.465174 19609 catalog_manager.cc:1259] Loaded cluster ID: 483357c80b9446be8a5b58e833a5645b
I20260929 01:10:29.465278 19609 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e: loading cluster ID for follower catalog manager: success
I20260929 01:10:29.467768 19609 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e: acquiring CA information for follower catalog manager: success
I20260929 01:10:29.469529 19609 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5488f70e07754a57ba2be20f6e75b96e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:10:29.815272 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 19356
I20260929 01:10:29.816686 19418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:29.933940 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 19356
I20260929 01:10:29.955560 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 19430
I20260929 01:10:29.956785 19492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:30.064488 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 19430
W20260929 01:10:30.085551 19523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.198.253:45751 (error 108)
I20260929 01:10:30.085726 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 19504
I20260929 01:10:30.086971 19566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:30.196122 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:30.086886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:45751 (local address 127.0.0.1:45636)
0929 01:10:30.195686 (+108800us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:30.195859 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108688,"thread_start_us":108583,"threads_started":1}
I20260929 01:10:30.196169 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 19504
2026-09-29T01:10:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4730 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:10:30 dist-test-slave-41cl krb5kdc[19648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:10:30 dist-test-slave-41cl krb5kdc[19648](info): set up 2 sockets
Sep 29 01:10:30 dist-test-slave-41cl krb5kdc[19648](info): commencing operation
krb5kdc: starting...
W20260929 01:10:31.309120 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:10:31 dist-test-slave-41cl krb5kdc[19648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:10:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:10:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:10:31.548493 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:37579
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:37579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:10:31.880751 19661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:31.881089 19661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:31.881264 19661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:31.888599 19661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:10:31.888739 19661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:31.888844 19661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:31.888949 19661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:10:31.889016 19661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:10:31.899083 19661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:37579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:37579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:31.900576 19661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:31.901885 19661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:31.914723 19666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:31.914657 19669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:31.914951 19667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:31.915760 19661 server_base.cc:1034] running on GCE node
I20260929 01:10:31.916652 19661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:31.918198 19661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:31.919437 19661 hybrid_clock.cc:648] HybridClock initialized: now 1790644231919397 us; error 52 us; skew 500 ppm
I20260929 01:10:31.919782 19661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:31 dist-test-slave-41cl krb5kdc[19648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644231, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:31.929942 19661 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:10:31.931623 19661 webserver.cc:460] Webserver started at http://127.5.198.254:45289/ using document root <none> and password file <none>
I20260929 01:10:31.932319 19661 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:31.932507 19661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:31.932839 19661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:10:31.935847 19661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "cfdabf5d736344d0b2d497511fd11492"
format_stamp: "Formatted at 2026-09-29 01:10:31 on dist-test-slave-41cl"
server_key: "cb04c4f935884bb70fd9961b674019e5"
server_key_iv: "3aedbb2b53b8c4cae0b4bbb0912ca1d3"
server_key_version: "encryptionkey@0"
I20260929 01:10:31.936757 19661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "cfdabf5d736344d0b2d497511fd11492"
format_stamp: "Formatted at 2026-09-29 01:10:31 on dist-test-slave-41cl"
server_key: "cb04c4f935884bb70fd9961b674019e5"
server_key_iv: "3aedbb2b53b8c4cae0b4bbb0912ca1d3"
server_key_version: "encryptionkey@0"
I20260929 01:10:31.941936 19661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:10:31.945598 19676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:31.947008 19661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:10:31.947216 19661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "cfdabf5d736344d0b2d497511fd11492"
format_stamp: "Formatted at 2026-09-29 01:10:31 on dist-test-slave-41cl"
server_key: "cb04c4f935884bb70fd9961b674019e5"
server_key_iv: "3aedbb2b53b8c4cae0b4bbb0912ca1d3"
server_key_version: "encryptionkey@0"
I20260929 01:10:31.947427 19661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:31.984818 19661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:31.989079 19661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:31.989393 19661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:32.009311 19661 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:37579
I20260929 01:10:32.009359 19727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:37579 every 8 connection(s)
I20260929 01:10:32.010746 19661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:10:32.011468 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 19661
I20260929 01:10:32.011720 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:10:32.012528 5915 external_mini_cluster.cc:1428] Setting key e12eeed31fa2619d25f3bc314d6a33cf
I20260929 01:10:32.015442 19728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 01:10:32 dist-test-slave-41cl krb5kdc[19648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:32.028229 19728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492: Bootstrap starting.
I20260929 01:10:32.032397 19728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492: Neither blocks nor log segments found. Creating new log.
I20260929 01:10:32.033624 19728 log.cc:824] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:32.037277 19728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492: No bootstrap required, opened a new log
I20260929 01:10:32.037895 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:32.014586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.5.198.254:37579)
0929 01:10:32.015377 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:32.015394 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:32.015450 (+ 56us) server_negotiation.cc:407] Connection header received
0929 01:10:32.017773 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:32.017798 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:32.018280 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:32.018736 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:32.020154 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:32.022206 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:32.023558 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:32.024585 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:32.024770 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:32.028981 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:32.028993 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:32.029012 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:32.029061 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:32.033080 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:32.033820 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:32.033829 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:32.033836 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:32.034046 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:32.034584 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:32.034588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:32.034590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:32.035307 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:32.035603 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:32.036056 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 01:10:32.036526 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":339,"threads_started":1}
I20260929 01:10:32.041953 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:10:32.046001 19728 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdabf5d736344d0b2d497511fd11492" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37579 } }
I20260929 01:10:32.046381 19728 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:10:32.046499 19728 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfdabf5d736344d0b2d497511fd11492, State: Initialized, Role: FOLLOWER
I20260929 01:10:32.047106 19728 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdabf5d736344d0b2d497511fd11492" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37579 } }
I20260929 01:10:32.047341 19728 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:10:32.047470 19728 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:10:32.047640 19728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:10:32.049873 19728 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdabf5d736344d0b2d497511fd11492" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37579 } }
I20260929 01:10:32.050405 19728 leader_election.cc:302] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfdabf5d736344d0b2d497511fd11492; no voters:
I20260929 01:10:32.051033 19728 leader_election.cc:288] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:10:32.051146 19733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:10:32.051908 19733 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [term 1 LEADER]: Becoming Leader. State: Replica: cfdabf5d736344d0b2d497511fd11492, State: Running, Role: LEADER
I20260929 01:10:32.052507 19733 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdabf5d736344d0b2d497511fd11492" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37579 } }
I20260929 01:10:32.053272 19728 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:10:32.055572 19735 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfdabf5d736344d0b2d497511fd11492. Latest consensus state: current_term: 1 leader_uuid: "cfdabf5d736344d0b2d497511fd11492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdabf5d736344d0b2d497511fd11492" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37579 } } }
I20260929 01:10:32.055718 19734 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfdabf5d736344d0b2d497511fd11492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdabf5d736344d0b2d497511fd11492" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37579 } } }
I20260929 01:10:32.055998 19734 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:32.055998 19735 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:32.056941 19740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:10:32.064409 19740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:10:32.073292 19740 catalog_manager.cc:1347] Generated new cluster ID: a9968852aa954c17967333adc97421bc
I20260929 01:10:32.073412 19740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:10:32.085654 19740 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:10:32.087441 19740 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:10:32.096351 19740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfdabf5d736344d0b2d497511fd11492: Generated new TSK 0
I20260929 01:10:32.097455 19740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:10:32.154026 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 19661
I20260929 01:10:32.155305 19723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:32.253551 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 19661
2026-09-29T01:10:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2052 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:10:32 dist-test-slave-41cl krb5kdc[19756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:10:32 dist-test-slave-41cl krb5kdc[19756](info): set up 2 sockets
Sep 29 01:10:32 dist-test-slave-41cl krb5kdc[19756](info): commencing operation
krb5kdc: starting...
W20260929 01:10:33.357065 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:10:33 dist-test-slave-41cl krb5kdc[19756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:10:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:10:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:10:33.588644 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:33223
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:33223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:10:33.926038 19769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:33.926311 19769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:33.926499 19769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:33.933772 19769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:10:33.933876 19769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:33.933975 19769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:33.934093 19769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:10:33.934165 19769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:10:33.944206 19769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:33223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:33223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:33.945583 19769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:33.946877 19769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:33.959106 19777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:33.959268 19775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:33.959383 19774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:33.960152 19769 server_base.cc:1034] running on GCE node
I20260929 01:10:33.961050 19769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:33.962525 19769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:33.963727 19769 hybrid_clock.cc:648] HybridClock initialized: now 1790644233963684 us; error 31 us; skew 500 ppm
I20260929 01:10:33.964109 19769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:33 dist-test-slave-41cl krb5kdc[19756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644233, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:33.967403 19769 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:10:33.969084 19769 webserver.cc:460] Webserver started at http://127.5.198.254:45499/ using document root <none> and password file <none>
I20260929 01:10:33.969796 19769 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:33.969923 19769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:33.970243 19769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:10:33.973362 19769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "d352ec48235d4fdab189332b74e40023"
format_stamp: "Formatted at 2026-09-29 01:10:33 on dist-test-slave-41cl"
server_key: "0ebafa4debdbe547f3ce4fc73bbf82c5"
server_key_iv: "c86f8a7444c260873cbcb28bc31f433a"
server_key_version: "encryptionkey@0"
I20260929 01:10:33.974143 19769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "d352ec48235d4fdab189332b74e40023"
format_stamp: "Formatted at 2026-09-29 01:10:33 on dist-test-slave-41cl"
server_key: "0ebafa4debdbe547f3ce4fc73bbf82c5"
server_key_iv: "c86f8a7444c260873cbcb28bc31f433a"
server_key_version: "encryptionkey@0"
I20260929 01:10:33.979557 19769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:10:33.983462 19784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:33.985498 19769 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 01:10:33.985723 19769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "d352ec48235d4fdab189332b74e40023"
format_stamp: "Formatted at 2026-09-29 01:10:33 on dist-test-slave-41cl"
server_key: "0ebafa4debdbe547f3ce4fc73bbf82c5"
server_key_iv: "c86f8a7444c260873cbcb28bc31f433a"
server_key_version: "encryptionkey@0"
I20260929 01:10:33.985945 19769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:34.005366 19769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:34.009968 19769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:34.010281 19769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:34.029270 19769 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:33223
I20260929 01:10:34.029322 19835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:33223 every 8 connection(s)
I20260929 01:10:34.030746 19769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:10:34.032337 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 19769
I20260929 01:10:34.032564 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:10:34.033445 5915 external_mini_cluster.cc:1428] Setting key 2490d067c1f1cf6dd9e465ed1195a8ef
I20260929 01:10:34.035631 19836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 01:10:34 dist-test-slave-41cl krb5kdc[19756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:34.049463 19836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023: Bootstrap starting.
I20260929 01:10:34.052635 19836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023: Neither blocks nor log segments found. Creating new log.
I20260929 01:10:34.053653 19836 log.cc:824] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:34.056659 19836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023: No bootstrap required, opened a new log
I20260929 01:10:34.057176 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:34.035419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60782 (local address 127.5.198.254:33223)
0929 01:10:34.036180 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:34.036191 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:34.036231 (+ 40us) server_negotiation.cc:407] Connection header received
0929 01:10:34.037850 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:34.037872 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:34.038260 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:34.038598 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:34.040418 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:34.041833 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:34.043668 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:34.044426 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:34.044659 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:34.049724 (+ 5065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:34.049732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:34.049747 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:34.049785 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:34.052948 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:34.053693 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:34.053700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:34.053706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:34.053924 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:34.054324 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:34.054333 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:34.054336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:34.054865 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:34.055198 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:34.055654 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 01:10:34.056095 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":354,"threads_started":1}
I20260929 01:10:34.064308 19836 raft_consensus.cc:348] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } }
I20260929 01:10:34.064782 19836 raft_consensus.cc:374] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:10:34.064899 19836 raft_consensus.cc:729] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d352ec48235d4fdab189332b74e40023, State: Initialized, Role: FOLLOWER
I20260929 01:10:34.065595 19836 consensus_queue.cc:260] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } }
I20260929 01:10:34.065810 19836 raft_consensus.cc:388] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:10:34.065896 19836 raft_consensus.cc:482] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:10:34.066028 19836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:10:34.068109 19836 raft_consensus.cc:504] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } }
I20260929 01:10:34.068641 19836 leader_election.cc:302] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d352ec48235d4fdab189332b74e40023; no voters:
I20260929 01:10:34.069211 19836 leader_election.cc:288] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:10:34.069391 19841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:10:34.070819 19841 raft_consensus.cc:686] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 1 LEADER]: Becoming Leader. State: Replica: d352ec48235d4fdab189332b74e40023, State: Running, Role: LEADER
I20260929 01:10:34.071444 19836 sys_catalog.cc:564] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:10:34.071390 19841 consensus_queue.cc:237] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } }
I20260929 01:10:34.074491 19842 sys_catalog.cc:455] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d352ec48235d4fdab189332b74e40023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } } }
I20260929 01:10:34.074437 19843 sys_catalog.cc:455] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d352ec48235d4fdab189332b74e40023. Latest consensus state: current_term: 1 leader_uuid: "d352ec48235d4fdab189332b74e40023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } } }
I20260929 01:10:34.074836 19842 sys_catalog.cc:458] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:34.074880 19843 sys_catalog.cc:458] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:34.077229 19850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:10:34.084192 19850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:10:34.093712 19850 catalog_manager.cc:1347] Generated new cluster ID: 527a6028225e47d1835e808cb5fcb580
I20260929 01:10:34.093849 19850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:10:34.117512 19850 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:10:34.119481 19850 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:10:34.126467 19850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023: Generated new TSK 0
I20260929 01:10:34.127436 19850 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:10:34.203763 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33223
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:10:34.567819 19862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:34.568138 19862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:34.568327 19862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:34.575883 19862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:34.576040 19862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:34.576212 19862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:10:34.586412 19862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:34.587772 19862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:34.589205 19862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:34.602525 19867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:34.602485 19868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:34.602442 19870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:34.603484 19862 server_base.cc:1034] running on GCE node
I20260929 01:10:34.604140 19862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:34.605293 19862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:34.606527 19862 hybrid_clock.cc:648] HybridClock initialized: now 1790644234606489 us; error 75 us; skew 500 ppm
I20260929 01:10:34.606920 19862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:34 dist-test-slave-41cl krb5kdc[19756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644234, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:34.610563 19862 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:10:34.612231 19862 webserver.cc:460] Webserver started at http://127.5.198.193:46515/ using document root <none> and password file <none>
I20260929 01:10:34.613039 19862 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:34.613189 19862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:34.613500 19862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:10:34.616426 19862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "0b173f6af4bc48168d80a0b064e3b7b9"
format_stamp: "Formatted at 2026-09-29 01:10:34 on dist-test-slave-41cl"
server_key: "ccb0e4e25b83f8362066dc7358e362ad"
server_key_iv: "86e1a775a1dfbbdc3c801956561ac8f8"
server_key_version: "encryptionkey@0"
I20260929 01:10:34.617259 19862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "0b173f6af4bc48168d80a0b064e3b7b9"
format_stamp: "Formatted at 2026-09-29 01:10:34 on dist-test-slave-41cl"
server_key: "ccb0e4e25b83f8362066dc7358e362ad"
server_key_iv: "86e1a775a1dfbbdc3c801956561ac8f8"
server_key_version: "encryptionkey@0"
I20260929 01:10:34.622540 19862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:10:34.626242 19877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:34.627676 19862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:10:34.627854 19862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "0b173f6af4bc48168d80a0b064e3b7b9"
format_stamp: "Formatted at 2026-09-29 01:10:34 on dist-test-slave-41cl"
server_key: "ccb0e4e25b83f8362066dc7358e362ad"
server_key_iv: "86e1a775a1dfbbdc3c801956561ac8f8"
server_key_version: "encryptionkey@0"
I20260929 01:10:34.628046 19862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:34.651225 19862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:34.655553 19862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:34.655846 19862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:34.656884 19862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:34.658192 19862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:10:34.658265 19862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:34.658386 19862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:10:34.658448 19862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:34.692032 19862 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:32943
I20260929 01:10:34.692123 19989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:32943 every 8 connection(s)
I20260929 01:10:34.693326 19862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:10:34.700793 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 19862
I20260929 01:10:34.700999 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:10:34.701635 5915 external_mini_cluster.cc:1428] Setting key e69acec871a9d21c0a4cf65972c94887
Sep 29 01:10:34 dist-test-slave-41cl krb5kdc[19756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644234, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:34.715308 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:34.695835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:56347 (local address 127.5.198.254:33223)
0929 01:10:34.696292 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:34.696296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:34.697303 (+ 1007us) server_negotiation.cc:407] Connection header received
0929 01:10:34.699308 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:34.699315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:34.699420 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:34.699550 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:34.701286 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:34.702134 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:34.704034 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:34.704458 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:34.704663 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:34.708653 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:34.708659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:34.708663 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:34.708700 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:34.711492 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:34.712686 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:34.712690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:34.712693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:34.712829 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:34.713357 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:34.713361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:34.713363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:34.713632 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:34.713744 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:34.714723 (+ 979us) server_negotiation.cc:299] Negotiation successful
0929 01:10:34.714946 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":253,"threads_started":1}
I20260929 01:10:34.716970 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:34.696258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:56347)
0929 01:10:34.697121 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:34.697173 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:34.699010 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:34.699794 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:34.699802 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:34.700331 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:34.701097 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:34.701130 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:34.702305 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:34.702311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:34.703856 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:34.703869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:34.704745 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:34.704755 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:34.705019 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:34.705758 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:34.705781 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:34.708366 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:34.711652 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:34.711665 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:34.711689 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:34.712541 (+ 852us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:34.712954 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:34.712962 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:34.712969 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:34.713235 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:34.713762 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:34.713773 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:34.714267 (+ 494us) client_negotiation.cc:769] Sending connection context
0929 01:10:34.714615 (+ 348us) client_negotiation.cc:240] Negotiation successful
0929 01:10:34.715251 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":375,"threads_started":1}
I20260929 01:10:34.718662 19990 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33223
I20260929 01:10:34.719009 19990 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:34.719736 19990 heartbeater.cc:511] Master 127.5.198.254:33223 requested a full tablet report, sending...
I20260929 01:10:34.721985 19801 ts_manager.cc:194] Registered new tserver with Master: 0b173f6af4bc48168d80a0b064e3b7b9 (127.5.198.193:32943)
I20260929 01:10:34.723475 19801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:56347
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:10:34.760800 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33223
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:10:35.084436 19996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:35.084827 19996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:35.084995 19996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:35.092597 19996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:35.092739 19996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:35.092932 19996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:10:35.102730 19996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:35.104061 19996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:35.105479 19996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:35.118453 20004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:35.118537 20002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:35.118562 20001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:35.119565 19996 server_base.cc:1034] running on GCE node
I20260929 01:10:35.120157 19996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:35.121201 19996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:35.122419 19996 hybrid_clock.cc:648] HybridClock initialized: now 1790644235122343 us; error 63 us; skew 500 ppm
I20260929 01:10:35.122746 19996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:35 dist-test-slave-41cl krb5kdc[19756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644235, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:35.126216 19996 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:10:35.127764 19996 webserver.cc:460] Webserver started at http://127.5.198.194:45875/ using document root <none> and password file <none>
I20260929 01:10:35.128437 19996 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:35.128618 19996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:35.128930 19996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:10:35.131903 19996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "f02510081e38460ca54fbef2a3dd3983"
format_stamp: "Formatted at 2026-09-29 01:10:35 on dist-test-slave-41cl"
server_key: "650f214e68c026ad44a28a70351c9bf6"
server_key_iv: "050f9944557659bf6e1e7b10d11ba925"
server_key_version: "encryptionkey@0"
I20260929 01:10:35.132757 19996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "f02510081e38460ca54fbef2a3dd3983"
format_stamp: "Formatted at 2026-09-29 01:10:35 on dist-test-slave-41cl"
server_key: "650f214e68c026ad44a28a70351c9bf6"
server_key_iv: "050f9944557659bf6e1e7b10d11ba925"
server_key_version: "encryptionkey@0"
I20260929 01:10:35.137789 19996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:10:35.141129 20011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:35.142517 19996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:10:35.142649 19996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "f02510081e38460ca54fbef2a3dd3983"
format_stamp: "Formatted at 2026-09-29 01:10:35 on dist-test-slave-41cl"
server_key: "650f214e68c026ad44a28a70351c9bf6"
server_key_iv: "050f9944557659bf6e1e7b10d11ba925"
server_key_version: "encryptionkey@0"
I20260929 01:10:35.142808 19996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:35.158314 19996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:35.162652 19996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:35.162892 19996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:35.163782 19996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:35.165200 19996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:10:35.165272 19996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:35.165354 19996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:10:35.165401 19996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:35.202310 19996 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:34065
I20260929 01:10:35.202349 20123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:34065 every 8 connection(s)
I20260929 01:10:35.203665 19996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:10:35.212985 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 19996
I20260929 01:10:35.213197 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:10:35.213613 5915 external_mini_cluster.cc:1428] Setting key 4f250b6442ea0c876e88a05a1f36b1dc
Sep 29 01:10:35 dist-test-slave-41cl krb5kdc[19756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644235, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:35.225430 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:35.206336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:40631 (local address 127.5.198.254:33223)
0929 01:10:35.206583 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:35.206588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:35.207676 (+ 1088us) server_negotiation.cc:407] Connection header received
0929 01:10:35.209573 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:35.209577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:35.209647 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:35.209736 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:35.211188 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.211917 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.213860 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.214462 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.214600 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:35.218538 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:35.218546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:35.218554 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:35.218600 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:35.222291 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:35.223024 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:35.223033 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:35.223039 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:35.223198 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:35.223645 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:35.223652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:35.223657 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:35.224055 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:35.224224 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:35.225017 (+ 793us) server_negotiation.cc:299] Negotiation successful
0929 01:10:35.225189 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260929 01:10:35.226884 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:35.206791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:40631)
0929 01:10:35.207511 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:35.207547 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:35.209293 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:35.209958 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:35.209966 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:35.210463 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:35.211014 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:35.211031 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.212120 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.212124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:35.213682 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:35.213715 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.214696 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.214705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:35.214995 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:35.215786 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:35.215817 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:35.218285 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:35.222458 (+ 4173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:35.222465 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:35.222479 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:35.222888 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:35.223310 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:35.223313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:35.223315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:35.223514 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:35.224181 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:35.224187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:35.224587 (+ 400us) client_negotiation.cc:769] Sending connection context
0929 01:10:35.224901 (+ 314us) client_negotiation.cc:240] Negotiation successful
0929 01:10:35.225476 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":282,"threads_started":1}
I20260929 01:10:35.228582 20124 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33223
I20260929 01:10:35.228930 20124 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:35.229738 20124 heartbeater.cc:511] Master 127.5.198.254:33223 requested a full tablet report, sending...
I20260929 01:10:35.231359 19801 ts_manager.cc:194] Registered new tserver with Master: f02510081e38460ca54fbef2a3dd3983 (127.5.198.194:34065)
I20260929 01:10:35.232647 19801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:40631
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:10:35.269279 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33223
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:10:35.604275 20129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:35.604691 20129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:35.604887 20129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:35.613533 20129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:35.613700 20129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:35.613934 20129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:10:35.624594 20129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:35.626135 20129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:35.627538 20129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:35.642351 20134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:35.642359 20135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:35.642351 20137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:35.643998 20129 server_base.cc:1034] running on GCE node
I20260929 01:10:35.644742 20129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:35.645711 20129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:35.646994 20129 hybrid_clock.cc:648] HybridClock initialized: now 1790644235646924 us; error 78 us; skew 500 ppm
I20260929 01:10:35.647531 20129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:35 dist-test-slave-41cl krb5kdc[19756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644235, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:35.651054 20129 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:10:35.652740 20129 webserver.cc:460] Webserver started at http://127.5.198.195:39191/ using document root <none> and password file <none>
I20260929 01:10:35.653487 20129 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:35.653645 20129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:35.653980 20129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:10:35.657210 20129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "05a057a9edb6460ea39722b23a8cae97"
format_stamp: "Formatted at 2026-09-29 01:10:35 on dist-test-slave-41cl"
server_key: "5d01832eccae62fd211499228a73868d"
server_key_iv: "dcdd18ab5da7d35156ea2b37b01fa6bd"
server_key_version: "encryptionkey@0"
I20260929 01:10:35.658061 20129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "05a057a9edb6460ea39722b23a8cae97"
format_stamp: "Formatted at 2026-09-29 01:10:35 on dist-test-slave-41cl"
server_key: "5d01832eccae62fd211499228a73868d"
server_key_iv: "dcdd18ab5da7d35156ea2b37b01fa6bd"
server_key_version: "encryptionkey@0"
I20260929 01:10:35.663697 20129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 01:10:35.667249 20144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:35.668764 20129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:10:35.668987 20129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "05a057a9edb6460ea39722b23a8cae97"
format_stamp: "Formatted at 2026-09-29 01:10:35 on dist-test-slave-41cl"
server_key: "5d01832eccae62fd211499228a73868d"
server_key_iv: "dcdd18ab5da7d35156ea2b37b01fa6bd"
server_key_version: "encryptionkey@0"
I20260929 01:10:35.669201 20129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:35.696192 20129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:35.700580 20129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:35.700871 20129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:35.701834 20129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:35.703255 20129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:10:35.703362 20129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:35.703518 20129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:10:35.703591 20129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:35.726713 19990 heartbeater.cc:503] Master 127.5.198.254:33223 was elected leader, sending a full tablet report...
I20260929 01:10:35.738190 20129 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:46321
I20260929 01:10:35.738235 20256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:46321 every 8 connection(s)
I20260929 01:10:35.739449 20129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:10:35.743172 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 20129
I20260929 01:10:35.743368 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:10:35.743897 5915 external_mini_cluster.cc:1428] Setting key 772ba904e68448d70b3eb308a059aca7
Sep 29 01:10:35 dist-test-slave-41cl krb5kdc[19756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644235, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:35.759471 20259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:35.742064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:45559 (local address 127.5.198.254:33223)
0929 01:10:35.742520 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:35.742525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:35.743647 (+ 1122us) server_negotiation.cc:407] Connection header received
0929 01:10:35.745724 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:35.745730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:35.745822 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:35.745937 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:35.747640 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.748387 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.749904 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.750304 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.750454 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:35.753810 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:35.753815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:35.753819 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:35.753851 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:35.756417 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:35.757230 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:35.757235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:35.757241 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:35.757365 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:35.757783 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:35.757788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:35.757792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:35.758069 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:35.758224 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:35.758888 (+ 664us) server_negotiation.cc:299] Negotiation successful
0929 01:10:35.759095 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":242,"threads_started":1}
I20260929 01:10:35.760372 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:35.742639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:45559)
0929 01:10:35.743486 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:35.743535 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:35.745472 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:35.746168 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:35.746176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:35.746862 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:35.747438 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:35.747467 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.748561 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.748567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:35.749754 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:35.749766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.750519 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.750527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:35.750757 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:35.751417 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:35.751438 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:35.753609 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:35.756635 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:35.756643 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:35.756659 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:35.757103 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:35.757474 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:35.757480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:35.757483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:35.757664 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:35.758204 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:35.758211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:35.758518 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 01:10:35.758772 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 01:10:35.759292 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":334,"threads_started":1}
I20260929 01:10:35.761657 20257 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33223
I20260929 01:10:35.761965 20257 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:35.762648 20257 heartbeater.cc:511] Master 127.5.198.254:33223 requested a full tablet report, sending...
I20260929 01:10:35.764021 19801 ts_manager.cc:194] Registered new tserver with Master: 05a057a9edb6460ea39722b23a8cae97 (127.5.198.195:46321)
I20260929 01:10:35.764969 19801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:45559
I20260929 01:10:35.771093 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:10:35.788601 20259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:35.775841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60790 (local address 127.5.198.254:33223)
0929 01:10:35.776082 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:35.776086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:35.776411 (+ 325us) server_negotiation.cc:407] Connection header received
0929 01:10:35.776622 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:35.776625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:35.776690 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:35.776773 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:35.778090 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.778802 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.780031 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.780429 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.780614 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:35.783437 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:35.783442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:35.783445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:35.783479 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:35.785888 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:35.786615 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:35.786622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:35.786626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:35.786749 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:35.787125 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:35.787133 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:35.787136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:35.787477 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:35.787730 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:35.788037 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 01:10:35.788264 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 01:10:35.792826 19801 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:10:35.799732 19801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:10:35.833021 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:35.812361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.5.198.193:32943)
0929 01:10:35.823636 (+ 11275us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:35.823647 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:35.824192 (+ 545us) server_negotiation.cc:407] Connection header received
0929 01:10:35.824307 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:35.824317 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:35.824603 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:35.825659 (+ 1056us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:10:35.825953 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.827589 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.830037 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.831342 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.831517 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:35.831756 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:35.832385 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 01:10:35.832637 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11085,"thread_start_us":10915,"threads_started":1}
I20260929 01:10:35.833057 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:35.812145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:32943 (local address 127.0.0.1:57832)
0929 01:10:35.823907 (+ 11762us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:35.823924 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:35.824051 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:35.824880 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:35.824891 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:35.824946 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:10:35.825422 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:35.825436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.827739 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.827745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:35.829868 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:35.829884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.831877 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.831883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:35.832073 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:35.832123 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 01:10:35.832331 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 01:10:35.832583 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11651,"mutex_wait_us":82,"thread_start_us":11213,"threads_started":1}
I20260929 01:10:35.834314 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:35.811541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:46321 (local address 127.0.0.1:50564)
0929 01:10:35.822780 (+ 11239us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:35.822849 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:35.823111 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:35.826472 (+ 3361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:35.826478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:35.826514 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:10:35.826964 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:35.826979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.828906 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.828913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:35.831169 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:35.831190 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.833413 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.833422 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:35.833610 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:35.833644 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 01:10:35.833744 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 01:10:35.833881 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10867,"mutex_wait_us":1,"thread_start_us":12125,"threads_started":1}
I20260929 01:10:35.835565 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:35.824502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33010 (local address 127.5.198.194:34065)
0929 01:10:35.826021 (+ 1519us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:35.826030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:35.826858 (+ 828us) server_negotiation.cc:407] Connection header received
0929 01:10:35.827198 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:35.827207 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:35.827413 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:35.827617 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:10:35.828723 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.830480 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.833084 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.834211 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.834361 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:35.834549 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:35.835061 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 01:10:35.835225 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1367,"thread_start_us":1246,"threads_started":1}
I20260929 01:10:35.835906 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:35.811887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.5.198.195:46321)
0929 01:10:35.825812 (+ 13925us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:35.825821 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:35.825858 (+ 37us) server_negotiation.cc:407] Connection header received
0929 01:10:35.825959 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:35.825967 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:35.826179 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:35.826614 (+ 435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:10:35.827192 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.828763 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.831381 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.832870 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.833013 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:35.833269 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:35.834701 (+ 1432us) server_negotiation.cc:299] Negotiation successful
0929 01:10:35.834919 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13711,"thread_start_us":13527,"threads_started":1}
I20260929 01:10:35.836293 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:35.826081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:34065 (local address 127.0.0.1:33010)
0929 01:10:35.826714 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:35.826737 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:35.826981 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:35.827889 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:35.827893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:35.827921 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:10:35.828528 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:35.828544 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.830635 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.830640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:35.832881 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:35.832901 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.834680 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.834687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:35.834841 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:35.834870 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 01:10:35.835808 (+ 938us) client_negotiation.cc:240] Negotiation successful
0929 01:10:35.835944 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":2,"thread_start_us":301,"threads_started":1}
I20260929 01:10:35.837412 19925 tablet_service.cc:1467] Processing CreateTablet for tablet 804c6231e7c141f9b0b8838b4f770690 (DEFAULT_TABLE table=test-table [id=c4b0c87ca9a9493b8e0a210650b05b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:10:35.838902 19925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 804c6231e7c141f9b0b8838b4f770690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:10:35.839417 20059 tablet_service.cc:1467] Processing CreateTablet for tablet 804c6231e7c141f9b0b8838b4f770690 (DEFAULT_TABLE table=test-table [id=c4b0c87ca9a9493b8e0a210650b05b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:10:35.839780 20192 tablet_service.cc:1467] Processing CreateTablet for tablet 804c6231e7c141f9b0b8838b4f770690 (DEFAULT_TABLE table=test-table [id=c4b0c87ca9a9493b8e0a210650b05b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:10:35.840685 20059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 804c6231e7c141f9b0b8838b4f770690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:10:35.841156 20192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 804c6231e7c141f9b0b8838b4f770690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:10:35.852196 20276 tablet_bootstrap.cc:492] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983: Bootstrap starting.
I20260929 01:10:35.854235 20277 tablet_bootstrap.cc:492] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9: Bootstrap starting.
I20260929 01:10:35.858443 20277 tablet_bootstrap.cc:654] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9: Neither blocks nor log segments found. Creating new log.
I20260929 01:10:35.859010 20276 tablet_bootstrap.cc:654] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983: Neither blocks nor log segments found. Creating new log.
I20260929 01:10:35.859876 20277 log.cc:824] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:35.860833 20276 log.cc:824] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:35.865072 20276 tablet_bootstrap.cc:492] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983: No bootstrap required, opened a new log
I20260929 01:10:35.865450 20276 ts_tablet_manager.cc:1397] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260929 01:10:35.868573 20278 tablet_bootstrap.cc:492] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97: Bootstrap starting.
I20260929 01:10:35.872032 20278 tablet_bootstrap.cc:654] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97: Neither blocks nor log segments found. Creating new log.
I20260929 01:10:35.873476 20278 log.cc:824] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:35.875794 20277 tablet_bootstrap.cc:492] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9: No bootstrap required, opened a new log
I20260929 01:10:35.876104 20277 ts_tablet_manager.cc:1397] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.000s
I20260929 01:10:35.876204 20278 tablet_bootstrap.cc:492] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97: No bootstrap required, opened a new log
I20260929 01:10:35.876507 20278 ts_tablet_manager.cc:1397] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20260929 01:10:35.878685 20276 raft_consensus.cc:348] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } }
I20260929 01:10:35.879225 20276 raft_consensus.cc:374] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:10:35.879350 20276 raft_consensus.cc:729] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02510081e38460ca54fbef2a3dd3983, State: Initialized, Role: FOLLOWER
I20260929 01:10:35.880116 20276 consensus_queue.cc:260] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } }
I20260929 01:10:35.884078 20278 raft_consensus.cc:348] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } }
I20260929 01:10:35.884436 20278 raft_consensus.cc:374] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:10:35.884574 20278 raft_consensus.cc:729] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a057a9edb6460ea39722b23a8cae97, State: Initialized, Role: FOLLOWER
I20260929 01:10:35.885279 20124 heartbeater.cc:503] Master 127.5.198.254:33223 was elected leader, sending a full tablet report...
I20260929 01:10:35.885278 20278 consensus_queue.cc:260] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } }
I20260929 01:10:35.886063 20276 ts_tablet_manager.cc:1428] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260929 01:10:35.886390 20257 heartbeater.cc:503] Master 127.5.198.254:33223 was elected leader, sending a full tablet report...
I20260929 01:10:35.888293 20278 ts_tablet_manager.cc:1428] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97: Time spent starting tablet: real 0.012s user 0.006s sys 0.003s
I20260929 01:10:35.889779 20277 raft_consensus.cc:348] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } }
I20260929 01:10:35.890286 20277 raft_consensus.cc:374] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:10:35.890411 20277 raft_consensus.cc:729] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b173f6af4bc48168d80a0b064e3b7b9, State: Initialized, Role: FOLLOWER
I20260929 01:10:35.891062 20277 consensus_queue.cc:260] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } }
I20260929 01:10:35.893025 20277 ts_tablet_manager.cc:1428] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9: Time spent starting tablet: real 0.017s user 0.009s sys 0.008s
W20260929 01:10:35.945683 19991 tablet.cc:2369] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:10:35.955760 20125 tablet.cc:2369] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:10:35.977182 20283 raft_consensus.cc:482] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:10:35.977486 20283 raft_consensus.cc:504] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } }
I20260929 01:10:35.979183 20283 leader_election.cc:288] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f02510081e38460ca54fbef2a3dd3983 (127.5.198.194:34065), 0b173f6af4bc48168d80a0b064e3b7b9 (127.5.198.193:32943)
I20260929 01:10:35.985857 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:35.979619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:58279 (local address 127.5.198.194:34065)
0929 01:10:35.979826 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:35.979832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:35.979855 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:10:35.980123 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:35.980128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:35.980214 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:35.980324 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:10:35.980777 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.981778 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.983902 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.985070 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.985183 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:35.985232 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:35.985580 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 01:10:35.985667 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:10:35.985972 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:35.979482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:34065 (local address 127.5.198.195:58279)
0929 01:10:35.979703 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:35.979726 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:35.979921 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:35.980358 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:35.980362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:35.980391 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:10:35.980637 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:35.980649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.981937 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.981943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:35.983746 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:35.983759 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.985220 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.985227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:35.985416 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:35.985450 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 01:10:35.985554 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 01:10:35.985687 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 01:10:35.986996 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:35.979482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:32943 (local address 127.5.198.195:53035)
0929 01:10:35.980371 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:35.980396 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:35.980555 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:35.980904 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:35.980908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:35.980933 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:10:35.981341 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:35.981352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.982706 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.982709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:35.984750 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:35.984765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.986362 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.986370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:35.986543 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:35.986574 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 01:10:35.986657 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 01:10:35.986737 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":166,"thread_start_us":375,"threads_started":1}
I20260929 01:10:35.987085 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:35.979673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:53035 (local address 127.5.198.193:32943)
0929 01:10:35.979855 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:35.979861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:35.980511 (+ 650us) server_negotiation.cc:407] Connection header received
0929 01:10:35.980691 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:35.980699 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:35.980794 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:35.980862 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:10:35.981530 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.982551 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.984915 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:35.986132 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:35.986281 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:35.986343 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:35.986716 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 01:10:35.986848 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:10:35.987085 20079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804c6231e7c141f9b0b8838b4f770690" candidate_uuid: "05a057a9edb6460ea39722b23a8cae97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02510081e38460ca54fbef2a3dd3983" is_pre_election: true
I20260929 01:10:35.987674 20079 raft_consensus.cc:2454] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05a057a9edb6460ea39722b23a8cae97 in term 0.
I20260929 01:10:35.988189 19945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804c6231e7c141f9b0b8838b4f770690" candidate_uuid: "05a057a9edb6460ea39722b23a8cae97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" is_pre_election: true
I20260929 01:10:35.988396 20148 leader_election.cc:302] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a057a9edb6460ea39722b23a8cae97, f02510081e38460ca54fbef2a3dd3983; no voters:
I20260929 01:10:35.988739 19945 raft_consensus.cc:2454] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05a057a9edb6460ea39722b23a8cae97 in term 0.
I20260929 01:10:35.988843 20283 raft_consensus.cc:2781] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:10:35.989035 20283 raft_consensus.cc:482] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:10:35.989158 20283 raft_consensus.cc:3037] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [term 0 FOLLOWER]: Advancing to term 1
W20260929 01:10:35.991107 20258 tablet.cc:2369] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:10:35.991968 20283 raft_consensus.cc:504] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } }
I20260929 01:10:35.992543 20283 leader_election.cc:288] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [CANDIDATE]: Term 1 election: Requested vote from peers f02510081e38460ca54fbef2a3dd3983 (127.5.198.194:34065), 0b173f6af4bc48168d80a0b064e3b7b9 (127.5.198.193:32943)
I20260929 01:10:35.992988 19945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804c6231e7c141f9b0b8838b4f770690" candidate_uuid: "05a057a9edb6460ea39722b23a8cae97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b173f6af4bc48168d80a0b064e3b7b9"
I20260929 01:10:35.993041 20079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804c6231e7c141f9b0b8838b4f770690" candidate_uuid: "05a057a9edb6460ea39722b23a8cae97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02510081e38460ca54fbef2a3dd3983"
I20260929 01:10:35.993167 19945 raft_consensus.cc:3037] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:10:35.993279 20079 raft_consensus.cc:3037] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:10:35.995308 19945 raft_consensus.cc:2454] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05a057a9edb6460ea39722b23a8cae97 in term 1.
I20260929 01:10:35.995468 20079 raft_consensus.cc:2454] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05a057a9edb6460ea39722b23a8cae97 in term 1.
I20260929 01:10:35.995705 20146 leader_election.cc:302] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a057a9edb6460ea39722b23a8cae97, 0b173f6af4bc48168d80a0b064e3b7b9; no voters:
I20260929 01:10:35.995988 20283 raft_consensus.cc:2781] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:10:35.996387 20283 raft_consensus.cc:686] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [term 1 LEADER]: Becoming Leader. State: Replica: 05a057a9edb6460ea39722b23a8cae97, State: Running, Role: LEADER
I20260929 01:10:35.996959 20283 consensus_queue.cc:237] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } }
I20260929 01:10:36.001650 19801 catalog_manager.cc:5495] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05a057a9edb6460ea39722b23a8cae97 (127.5.198.195). New cstate: current_term: 1 leader_uuid: "05a057a9edb6460ea39722b23a8cae97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } health_report { overall_health: UNKNOWN } } }
I20260929 01:10:36.034039 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:36.027767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50572 (local address 127.5.198.195:46321)
0929 01:10:36.028023 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:36.028027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:36.028045 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:10:36.028127 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:36.028131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:36.028206 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:36.028342 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:10:36.028822 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:36.029947 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:36.031499 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:36.032144 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:36.032282 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:36.032837 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:10:36.033471 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:10:36.033633 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:36.033695 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 01:10:36.033775 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 01:10:36.042227 20079 raft_consensus.cc:1261] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983 [term 1 FOLLOWER]: Refusing update from remote peer 05a057a9edb6460ea39722b23a8cae97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:10:36.042227 19945 raft_consensus.cc:1261] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Refusing update from remote peer 05a057a9edb6460ea39722b23a8cae97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:10:36.043217 20286 consensus_queue.cc:1035] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:10:36.043555 20283 consensus_queue.cc:1035] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:10:36.057164 20291 mvcc.cc:205] Tried to move back new op lower bound from 7334478790818525184 to 7334478790647164928. Current Snapshot: MvccSnapshot[applied={T|T < 7334478790818525184}]
I20260929 01:10:36.067041 19800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:10:36.067310 19800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:10:36.070984 19800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023: Sending DeleteTablet for 3 replicas of tablet 804c6231e7c141f9b0b8838b4f770690
I20260929 01:10:36.072081 19925 tablet_service.cc:1514] Processing DeleteTablet for tablet 804c6231e7c141f9b0b8838b4f770690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:10:36 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:57832
I20260929 01:10:36.072100 20192 tablet_service.cc:1514] Processing DeleteTablet for tablet 804c6231e7c141f9b0b8838b4f770690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:10:36 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:50564
I20260929 01:10:36.072099 20059 tablet_service.cc:1514] Processing DeleteTablet for tablet 804c6231e7c141f9b0b8838b4f770690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:10:36 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:33010
I20260929 01:10:36.073014 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 19862
I20260929 01:10:36.106426 20299 tablet_replica.cc:330] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97: stopping tablet replica
I20260929 01:10:36.107107 20299 raft_consensus.cc:2229] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:10:36.108002 20299 raft_consensus.cc:2258] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:10:36.109338 20301 tablet_replica.cc:330] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983: stopping tablet replica
I20260929 01:10:36.109783 20301 raft_consensus.cc:2229] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:10:36.110045 20300 tablet_replica.cc:330] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9: stopping tablet replica
I20260929 01:10:36.110409 20301 raft_consensus.cc:2258] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:10:36.110411 20300 raft_consensus.cc:2229] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:10:36.110857 20300 raft_consensus.cc:2258] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:10:36.112615 20301 ts_tablet_manager.cc:1905] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:10:36.112922 20300 ts_tablet_manager.cc:1905] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 01:10:36 dist-test-slave-41cl krb5kdc[19756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:10:36.119612 20301 ts_tablet_manager.cc:1918] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:10:36.119769 20301 log.cc:1196] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/804c6231e7c141f9b0b8838b4f770690
I20260929 01:10:36.119761 20300 ts_tablet_manager.cc:1918] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:10:36.119910 20300 log.cc:1196] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/804c6231e7c141f9b0b8838b4f770690
I20260929 01:10:36.120306 20301 ts_tablet_manager.cc:1939] T 804c6231e7c141f9b0b8838b4f770690 P f02510081e38460ca54fbef2a3dd3983: Deleting consensus metadata
I20260929 01:10:36.120430 20300 ts_tablet_manager.cc:1939] T 804c6231e7c141f9b0b8838b4f770690 P 0b173f6af4bc48168d80a0b064e3b7b9: Deleting consensus metadata
I20260929 01:10:36.121819 20299 ts_tablet_manager.cc:1905] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:10:36.122138 19788 catalog_manager.cc:4841] TS f02510081e38460ca54fbef2a3dd3983 (127.5.198.194:34065): tablet 804c6231e7c141f9b0b8838b4f770690 (table test-table [id=c4b0c87ca9a9493b8e0a210650b05b8b]) successfully deleted
I20260929 01:10:36.122252 19786 catalog_manager.cc:4841] TS 0b173f6af4bc48168d80a0b064e3b7b9 (127.5.198.193:32943): tablet 804c6231e7c141f9b0b8838b4f770690 (table test-table [id=c4b0c87ca9a9493b8e0a210650b05b8b]) successfully deleted
I20260929 01:10:36.124563 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:36.074381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57838 (local address 127.5.198.193:32943)
0929 01:10:36.074596 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:36.074605 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:36.097252 (+ 22647us) server_negotiation.cc:407] Connection header received
0929 01:10:36.097624 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:36.097633 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:36.097748 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:36.097888 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:36.100758 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:36.102364 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:36.108828 (+ 6464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:36.109555 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:36.109695 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:36.116232 (+ 6537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:36.116246 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:36.116267 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:36.116306 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:36.120752 (+ 4446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:36.121671 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:36.121679 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:36.121683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:36.121846 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:36.122499 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:36.122506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:36.122512 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:36.123009 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:36.123287 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:36.123945 (+ 658us) server_negotiation.cc:299] Negotiation successful
0929 01:10:36.124169 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 01:10:36.125191 19985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:36.129897 20299 ts_tablet_manager.cc:1918] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:10:36.130081 20299 log.cc:1196] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/804c6231e7c141f9b0b8838b4f770690
I20260929 01:10:36.130602 20299 ts_tablet_manager.cc:1939] T 804c6231e7c141f9b0b8838b4f770690 P 05a057a9edb6460ea39722b23a8cae97: Deleting consensus metadata
I20260929 01:10:36.132128 19788 catalog_manager.cc:4841] TS 05a057a9edb6460ea39722b23a8cae97 (127.5.198.195:46321): tablet 804c6231e7c141f9b0b8838b4f770690 (table test-table [id=c4b0c87ca9a9493b8e0a210650b05b8b]) successfully deleted
I20260929 01:10:36.255981 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 19862
I20260929 01:10:36.274919 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 19996
Sep 29 01:10:36 dist-test-slave-41cl krb5kdc[19756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:10:36.292603 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:36.275744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.5.198.194:34065)
0929 01:10:36.275933 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:36.275939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:36.275961 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:10:36.276145 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:36.276151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:36.276236 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:36.276363 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:36.277735 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:36.278918 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:36.280297 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:36.280955 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:36.281076 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:36.285427 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:36.285472 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:36.285501 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:36.285545 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:36.289266 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:36.290157 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:36.290169 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:36.290173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:36.290348 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:36.290811 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:36.290817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:36.290820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:36.291317 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:36.291631 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:36.291971 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 01:10:36.292196 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:10:36.293270 20119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:36.384014 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 19996
I20260929 01:10:36.407390 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 20129
Sep 29 01:10:36 dist-test-slave-41cl krb5kdc[19756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:10:36.424578 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:36.408354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.5.198.195:46321)
0929 01:10:36.408581 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:36.408586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:36.408607 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:10:36.408737 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:36.408741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:36.408844 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:36.408959 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:36.410180 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:36.410969 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:36.412199 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:36.412803 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:36.412952 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:36.417786 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:36.417800 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:36.417826 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:36.417873 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:36.421553 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:36.422413 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:36.422420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:36.422423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:36.422580 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:36.422999 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:36.423003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:36.423005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:36.423372 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:36.423649 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:36.423975 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 01:10:36.424155 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 01:10:36.425442 20252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:36.547701 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 20129
I20260929 01:10:36.567173 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 19769
I20260929 01:10:36.568332 19831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:36.682798 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 19769
I20260929 01:10:36.704527 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:33223
--webserver_interface=127.5.198.254
--webserver_port=45499
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:33223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:10:37.030189 20307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:37.030478 20307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:37.030570 20307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:37.037783 20307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:10:37.037891 20307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:37.037938 20307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:37.037994 20307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:10:37.038039 20307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:10:37.047670 20307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:33223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:33223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:37.048907 20307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:37.050114 20307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:37.073129 20313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:37.073093 20314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:37.073156 20316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:37.074213 20307 server_base.cc:1034] running on GCE node
I20260929 01:10:37.075038 20307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:37.076727 20307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:37.077968 20307 hybrid_clock.cc:648] HybridClock initialized: now 1790644237077890 us; error 41 us; skew 500 ppm
I20260929 01:10:37.078393 20307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:37 dist-test-slave-41cl krb5kdc[19756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:37.088708 20307 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:10:37.090556 20307 webserver.cc:460] Webserver started at http://127.5.198.254:45499/ using document root <none> and password file <none>
I20260929 01:10:37.091352 20307 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:37.091517 20307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:37.097746 20307 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:10:37.100978 20323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:37.102337 20307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:10:37.102533 20307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "d352ec48235d4fdab189332b74e40023"
format_stamp: "Formatted at 2026-09-29 01:10:33 on dist-test-slave-41cl"
server_key: "0ebafa4debdbe547f3ce4fc73bbf82c5"
server_key_iv: "c86f8a7444c260873cbcb28bc31f433a"
server_key_version: "encryptionkey@0"
I20260929 01:10:37.103670 20307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:37.135893 20307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:37.140451 20307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:37.140787 20307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:37.159986 20307 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:33223
I20260929 01:10:37.160019 20374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:33223 every 8 connection(s)
I20260929 01:10:37.161485 20307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:10:37.167333 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 20307
I20260929 01:10:37.168365 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:32943
--local_ip_for_outbound_sockets=127.5.198.193
--tserver_master_addrs=127.5.198.254:33223
--webserver_port=46515
--webserver_interface=127.5.198.193
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:10:37.168872 20375 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:10:37.174687 20375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023: Bootstrap starting.
I20260929 01:10:37.195353 20375 log.cc:824] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:37.207629 20375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:10:37.208179 20375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023: Bootstrap complete.
I20260929 01:10:37.217938 20375 raft_consensus.cc:348] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } }
I20260929 01:10:37.218849 20375 raft_consensus.cc:729] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d352ec48235d4fdab189332b74e40023, State: Initialized, Role: FOLLOWER
I20260929 01:10:37.219493 20375 consensus_queue.cc:260] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } }
I20260929 01:10:37.219753 20375 raft_consensus.cc:388] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:10:37.219895 20375 raft_consensus.cc:482] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:10:37.220078 20375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:10:37.223985 20375 raft_consensus.cc:504] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } }
I20260929 01:10:37.224596 20375 leader_election.cc:302] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d352ec48235d4fdab189332b74e40023; no voters:
I20260929 01:10:37.225253 20375 leader_election.cc:288] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 01:10:37.225387 20379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:10:37.226653 20379 raft_consensus.cc:686] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 2 LEADER]: Becoming Leader. State: Replica: d352ec48235d4fdab189332b74e40023, State: Running, Role: LEADER
I20260929 01:10:37.227309 20379 consensus_queue.cc:237] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } }
I20260929 01:10:37.227718 20375 sys_catalog.cc:564] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:10:37.229722 20381 sys_catalog.cc:455] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d352ec48235d4fdab189332b74e40023. Latest consensus state: current_term: 2 leader_uuid: "d352ec48235d4fdab189332b74e40023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } } }
I20260929 01:10:37.230203 20381 sys_catalog.cc:458] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:37.230785 20380 sys_catalog.cc:455] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d352ec48235d4fdab189332b74e40023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } } }
I20260929 01:10:37.231122 20380 sys_catalog.cc:458] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:37.237258 20385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:10:37.245390 20385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:10:37.246806 20385 catalog_manager.cc:1259] Loaded cluster ID: 527a6028225e47d1835e808cb5fcb580
I20260929 01:10:37.246913 20385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:10:37.250049 20385 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 01:10:37.268350 20385 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 01:10:37.281824 20385 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:10:37.283594 20385 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023: Loaded TSK: 0
I20260929 01:10:37.285043 20385 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 01:10:37.526595 20377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:37.526865 20377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:37.527009 20377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:37.534636 20377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:37.534740 20377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:37.534904 20377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:10:37.545028 20377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:32943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=46515
--tserver_master_addrs=127.5.198.254:33223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:37.546343 20377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:37.547668 20377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:37.562489 20403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:37.562453 20404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:37.563081 20406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:37.564415 20377 server_base.cc:1034] running on GCE node
I20260929 01:10:37.565124 20377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:37.566165 20377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:37.567399 20377 hybrid_clock.cc:648] HybridClock initialized: now 1790644237567357 us; error 46 us; skew 500 ppm
I20260929 01:10:37.567727 20377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:37 dist-test-slave-41cl krb5kdc[19756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:37.570943 20377 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:10:37.572540 20377 webserver.cc:460] Webserver started at http://127.5.198.193:46515/ using document root <none> and password file <none>
I20260929 01:10:37.573299 20377 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:37.573446 20377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:37.578894 20377 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:10:37.581691 20413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:37.583159 20377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:10:37.583315 20377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "0b173f6af4bc48168d80a0b064e3b7b9"
format_stamp: "Formatted at 2026-09-29 01:10:34 on dist-test-slave-41cl"
server_key: "ccb0e4e25b83f8362066dc7358e362ad"
server_key_iv: "86e1a775a1dfbbdc3c801956561ac8f8"
server_key_version: "encryptionkey@0"
I20260929 01:10:37.584254 20377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:37.617945 20377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:37.622581 20377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:37.622869 20377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:37.623992 20377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:37.625408 20377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:10:37.625505 20377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:37.625645 20377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:10:37.625727 20377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:37.660251 20377 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:32943
I20260929 01:10:37.660409 20525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:32943 every 8 connection(s)
I20260929 01:10:37.661744 20377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:10:37.664683 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 20377
I20260929 01:10:37.665429 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:34065
--local_ip_for_outbound_sockets=127.5.198.194
--tserver_master_addrs=127.5.198.254:33223
--webserver_port=45875
--webserver_interface=127.5.198.194
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:10:37 dist-test-slave-41cl krb5kdc[19756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:37.687734 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:37.665291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:56401)
0929 01:10:37.666331 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:37.666377 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:37.668324 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:37.669946 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:37.669959 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:37.670638 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:37.671220 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:37.671240 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:37.673321 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:37.673328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:37.674904 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:37.674919 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:37.676120 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:37.676129 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:37.676442 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:37.677341 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:37.677369 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:37.679945 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:37.683426 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:37.683435 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:37.683449 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:37.683872 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:37.684302 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:37.684306 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:37.684313 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:37.684549 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:37.685268 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:37.685276 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:37.685608 (+ 332us) client_negotiation.cc:769] Sending connection context
0929 01:10:37.685884 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 01:10:37.686424 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":490,"threads_started":1}
I20260929 01:10:37.687721 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:37.665097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:56401 (local address 127.5.198.254:33223)
0929 01:10:37.665999 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:37.666018 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:37.666525 (+ 507us) server_negotiation.cc:407] Connection header received
0929 01:10:37.668991 (+ 2466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:37.669007 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:37.669373 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:37.669770 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:37.671496 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:37.673122 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:37.675082 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:37.675840 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:37.676085 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:37.680249 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:37.680257 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:37.680272 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:37.680317 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:37.683261 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:37.684024 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:37.684031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:37.684036 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:37.684183 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:37.684666 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:37.684673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:37.684676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:37.685124 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:37.685351 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:37.686085 (+ 734us) server_negotiation.cc:299] Negotiation successful
0929 01:10:37.686474 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":304,"threads_started":1}
I20260929 01:10:37.691105 20526 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33223
I20260929 01:10:37.691511 20526 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:37.692610 20526 heartbeater.cc:511] Master 127.5.198.254:33223 requested a full tablet report, sending...
I20260929 01:10:37.695025 20340 ts_manager.cc:194] Registered new tserver with Master: 0b173f6af4bc48168d80a0b064e3b7b9 (127.5.198.193:32943)
I20260929 01:10:37.696635 20340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:56401
W20260929 01:10:37.998385 20530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:37.998648 20530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:37.998793 20530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:38.006110 20530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:38.006227 20530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:38.006402 20530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:10:38.016307 20530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:34065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=45875
--tserver_master_addrs=127.5.198.254:33223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:38.017722 20530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:38.019039 20530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:38.032148 20535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:38.032123 20536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:38.032644 20538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:38.033099 20530 server_base.cc:1034] running on GCE node
I20260929 01:10:38.033648 20530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:38.034562 20530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:38.035751 20530 hybrid_clock.cc:648] HybridClock initialized: now 1790644238035717 us; error 24 us; skew 500 ppm
I20260929 01:10:38.036043 20530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:38 dist-test-slave-41cl krb5kdc[19756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644238, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:38.039157 20530 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:10:38.040609 20530 webserver.cc:460] Webserver started at http://127.5.198.194:45875/ using document root <none> and password file <none>
I20260929 01:10:38.041239 20530 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:38.041345 20530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:38.046532 20530 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:10:38.049830 20545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:38.051285 20530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:10:38.051420 20530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "f02510081e38460ca54fbef2a3dd3983"
format_stamp: "Formatted at 2026-09-29 01:10:35 on dist-test-slave-41cl"
server_key: "650f214e68c026ad44a28a70351c9bf6"
server_key_iv: "050f9944557659bf6e1e7b10d11ba925"
server_key_version: "encryptionkey@0"
I20260929 01:10:38.052332 20530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:38.064610 20530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:38.069434 20530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:38.069710 20530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:38.070724 20530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:38.072176 20530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:10:38.072290 20530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:38.072438 20530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:10:38.072613 20530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 01:10:38.112838 20530 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:34065
I20260929 01:10:38.112883 20657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:34065 every 8 connection(s)
I20260929 01:10:38.114545 20530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:10:38.119143 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 20530
I20260929 01:10:38.123754 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:46321
--local_ip_for_outbound_sockets=127.5.198.195
--tserver_master_addrs=127.5.198.254:33223
--webserver_port=39191
--webserver_interface=127.5.198.195
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:10:38 dist-test-slave-41cl krb5kdc[19756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644238, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:38.135257 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.117367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:42907 (local address 127.5.198.254:33223)
0929 01:10:38.117546 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:38.117551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:38.118387 (+ 836us) server_negotiation.cc:407] Connection header received
0929 01:10:38.120335 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:38.120341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:38.120439 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:38.120579 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:38.122070 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.122765 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.124453 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.124922 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.125057 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.129109 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:38.129115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:38.129119 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:38.129155 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:38.132078 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:38.132916 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:38.132925 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:38.132930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:38.133078 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:38.133513 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:38.133520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:38.133523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:38.133794 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:38.133959 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:38.134666 (+ 707us) server_negotiation.cc:299] Negotiation successful
0929 01:10:38.134903 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:10:38.136284 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.117529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:42907)
0929 01:10:38.118211 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:38.118247 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:38.120062 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:38.120833 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:38.120843 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:38.121334 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:38.121894 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:38.121912 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.122944 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.122949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:38.124288 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:38.124301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.125161 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.125168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:38.125407 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.126196 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:38.126221 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:38.128855 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:38.132253 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:38.132262 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:38.132284 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:38.132782 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:38.133210 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:38.133213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:38.133216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:38.133388 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:38.133941 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:38.133949 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:38.134289 (+ 340us) client_negotiation.cc:769] Sending connection context
0929 01:10:38.134573 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 01:10:38.135132 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":264,"threads_started":1}
I20260929 01:10:38.137822 20658 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33223
I20260929 01:10:38.138191 20658 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:38.139113 20658 heartbeater.cc:511] Master 127.5.198.254:33223 requested a full tablet report, sending...
I20260929 01:10:38.140810 20340 ts_manager.cc:194] Registered new tserver with Master: f02510081e38460ca54fbef2a3dd3983 (127.5.198.194:34065)
I20260929 01:10:38.141661 20340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:42907
W20260929 01:10:38.455646 20661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:38.455962 20661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:38.456123 20661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:38.463711 20661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:38.463832 20661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:38.464040 20661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:10:38.474511 20661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:46321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=39191
--tserver_master_addrs=127.5.198.254:33223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:38.475813 20661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:38.477242 20661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:38.490402 20666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:38.490330 20667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:38.490851 20669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:38.491302 20661 server_base.cc:1034] running on GCE node
I20260929 01:10:38.491909 20661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:38.493017 20661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:38.494271 20661 hybrid_clock.cc:648] HybridClock initialized: now 1790644238494234 us; error 29 us; skew 500 ppm
I20260929 01:10:38.494626 20661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:38 dist-test-slave-41cl krb5kdc[19756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644238, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:38.497893 20661 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:10:38.499219 20661 webserver.cc:460] Webserver started at http://127.5.198.195:39191/ using document root <none> and password file <none>
I20260929 01:10:38.499862 20661 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:38.499974 20661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:38.505348 20661 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:10:38.508239 20676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:38.509982 20661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:10:38.510162 20661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "05a057a9edb6460ea39722b23a8cae97"
format_stamp: "Formatted at 2026-09-29 01:10:35 on dist-test-slave-41cl"
server_key: "5d01832eccae62fd211499228a73868d"
server_key_iv: "dcdd18ab5da7d35156ea2b37b01fa6bd"
server_key_version: "encryptionkey@0"
I20260929 01:10:38.511109 20661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:38.538795 20661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:38.543413 20661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:38.543751 20661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:38.545058 20661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:38.546535 20661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:10:38.546634 20661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:38.546768 20661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:10:38.546851 20661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:38.581149 20661 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:46321
I20260929 01:10:38.581341 20788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:46321 every 8 connection(s)
I20260929 01:10:38.582587 20661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:10:38.589560 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 20661
Sep 29 01:10:38 dist-test-slave-41cl krb5kdc[19756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644238, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:38.608001 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.585425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:38571 (local address 127.5.198.254:33223)
0929 01:10:38.585658 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:38.585665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:38.586914 (+ 1249us) server_negotiation.cc:407] Connection header received
0929 01:10:38.588987 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:38.588994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:38.589095 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:38.589190 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:38.591524 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.592715 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.594690 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.595165 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.595256 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.599992 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:38.599997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:38.600001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:38.600030 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:38.603838 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:38.604793 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:38.604801 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:38.604806 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:38.604993 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:38.605520 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:38.605523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:38.605525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:38.605903 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:38.606072 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:38.607200 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0929 01:10:38.607444 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 01:10:38.608402 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.590725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.5.198.254:33223)
0929 01:10:38.591820 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:38.591826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:38.591844 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:10:38.591891 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:38.591896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:38.591968 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:38.592080 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:38.593934 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.594677 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.596908 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.597332 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.597425 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.601034 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:38.601044 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:38.601048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:38.601102 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:38.604716 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:38.605766 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:38.605770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:38.605773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:38.605894 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:38.606543 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:38.606547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:38.606550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:38.606848 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:38.607117 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:38.607506 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 01:10:38.607711 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":988,"thread_start_us":874,"threads_started":1}
I20260929 01:10:38.609295 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.586024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:38571)
0929 01:10:38.586754 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:38.586791 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:38.588714 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:38.589502 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:38.589514 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:38.590398 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:38.591288 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:38.591320 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.592880 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.592890 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:38.594537 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:38.594549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.595444 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.595456 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:38.595779 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.596701 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:38.596734 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:38.599712 (+ 2978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:38.604038 (+ 4326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:38.604048 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:38.604064 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:38.604674 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:38.605124 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:38.605130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:38.605133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:38.605397 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:38.606133 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:38.606141 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:38.606732 (+ 591us) client_negotiation.cc:769] Sending connection context
0929 01:10:38.607069 (+ 337us) client_negotiation.cc:240] Negotiation successful
0929 01:10:38.607792 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":302,"threads_started":1}
I20260929 01:10:38.610813 20789 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33223
I20260929 01:10:38.611334 20789 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:38.612161 20789 heartbeater.cc:511] Master 127.5.198.254:33223 requested a full tablet report, sending...
I20260929 01:10:38.614404 20340 ts_manager.cc:194] Registered new tserver with Master: 05a057a9edb6460ea39722b23a8cae97 (127.5.198.195:46321)
I20260929 01:10:38.615813 20340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:38571
I20260929 01:10:38.624745 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:10:38.632194 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.625781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60810 (local address 127.5.198.254:33223)
0929 01:10:38.626002 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:38.626011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:38.626506 (+ 495us) server_negotiation.cc:407] Connection header received
0929 01:10:38.626793 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:38.626798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:38.626941 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:38.627102 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:10:38.627552 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.628351 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.630024 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.630432 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.630579 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.631068 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:10:38.631590 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:10:38.631757 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:38.631828 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 01:10:38.631914 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:10:38.633129 20340 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:10:38.640643 20340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:10:38.666990 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.652565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:32943 (local address 127.0.0.1:57854)
0929 01:10:38.655029 (+ 2464us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:38.655046 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:38.655160 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:38.656116 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:38.656128 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:38.657282 (+ 1154us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:10:38.657675 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:38.657686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.659652 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.659658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:38.662864 (+ 3206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:38.662879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.666083 (+ 3204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.666089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:38.666280 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.666331 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 01:10:38.666458 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 01:10:38.666642 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2322,"thread_start_us":1003,"threads_started":1}
I20260929 01:10:38.668045 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.653205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.5.198.193:32943)
0929 01:10:38.654785 (+ 1580us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:38.654795 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:38.655290 (+ 495us) server_negotiation.cc:407] Connection header received
0929 01:10:38.655399 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:38.655406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:38.655651 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:38.655849 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:10:38.657880 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.659474 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.663058 (+ 3584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.665554 (+ 2496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.665687 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.665926 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:38.667540 (+ 1614us) server_negotiation.cc:299] Negotiation successful
0929 01:10:38.667705 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1386,"thread_start_us":1224,"threads_started":1}
I20260929 01:10:38.671622 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.652387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50594 (local address 127.5.198.195:46321)
0929 01:10:38.653090 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:38.653099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:38.654092 (+ 993us) server_negotiation.cc:407] Connection header received
0929 01:10:38.654543 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:38.654551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:38.654824 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:38.655019 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:10:38.656180 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.657957 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.668817 (+ 10860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.670242 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.670355 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.670577 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:38.671171 (+ 594us) server_negotiation.cc:299] Negotiation successful
0929 01:10:38.671334 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":364,"threads_started":1}
I20260929 01:10:38.672556 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.652132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:46321 (local address 127.0.0.1:50594)
0929 01:10:38.653950 (+ 1818us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:38.653989 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:38.654280 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:38.655363 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:38.655370 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:38.655400 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:10:38.655933 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:38.655951 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.658134 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.658139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:38.668620 (+ 10481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:38.668635 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.670777 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.670783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:38.670967 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.671000 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:10:38.672034 (+ 1034us) client_negotiation.cc:240] Negotiation successful
0929 01:10:38.672173 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1563,"thread_start_us":2621,"threads_started":1}
I20260929 01:10:38.672240 20461 tablet_service.cc:1467] Processing CreateTablet for tablet d419cc058406429ab88e5502e61cbc84 (DEFAULT_TABLE table=test-table [id=9891632cc2994ce98edf3fc110005c27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:10:38.674562 20461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d419cc058406429ab88e5502e61cbc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:10:38.676755 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.655138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:34065 (local address 127.0.0.1:33036)
0929 01:10:38.667256 (+ 12118us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:38.667275 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:38.667388 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:38.669024 (+ 1636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:38.669032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:38.669071 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:10:38.669775 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:38.669793 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.673069 (+ 3276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.673074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:38.674564 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:38.674577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.675938 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.675945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:38.676124 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.676157 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:10:38.676287 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 01:10:38.676425 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12031,"thread_start_us":17610,"threads_started":1}
I20260929 01:10:38.676841 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.655412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.5.198.194:34065)
0929 01:10:38.668019 (+ 12607us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:38.668031 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:38.668068 (+ 37us) server_negotiation.cc:407] Connection header received
0929 01:10:38.668198 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:38.668209 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:38.668528 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:38.670112 (+ 1584us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:10:38.670403 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.672407 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.674767 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.675722 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.675933 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.676142 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:38.676327 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 01:10:38.676516 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12381,"thread_start_us":13295,"threads_started":1}
I20260929 01:10:38.676819 20724 tablet_service.cc:1467] Processing CreateTablet for tablet d419cc058406429ab88e5502e61cbc84 (DEFAULT_TABLE table=test-table [id=9891632cc2994ce98edf3fc110005c27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:10:38.678303 20724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d419cc058406429ab88e5502e61cbc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:10:38.680255 20593 tablet_service.cc:1467] Processing CreateTablet for tablet d419cc058406429ab88e5502e61cbc84 (DEFAULT_TABLE table=test-table [id=9891632cc2994ce98edf3fc110005c27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:10:38.681286 20593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d419cc058406429ab88e5502e61cbc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:10:38.690595 20805 tablet_bootstrap.cc:492] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9: Bootstrap starting.
I20260929 01:10:38.693701 20806 tablet_bootstrap.cc:492] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97: Bootstrap starting.
I20260929 01:10:38.694659 20807 tablet_bootstrap.cc:492] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983: Bootstrap starting.
I20260929 01:10:38.696487 20805 tablet_bootstrap.cc:654] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9: Neither blocks nor log segments found. Creating new log.
I20260929 01:10:38.697265 20806 tablet_bootstrap.cc:654] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97: Neither blocks nor log segments found. Creating new log.
I20260929 01:10:38.697607 20805 log.cc:824] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:38.698060 20807 tablet_bootstrap.cc:654] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983: Neither blocks nor log segments found. Creating new log.
I20260929 01:10:38.698473 20806 log.cc:824] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:38.699368 20807 log.cc:824] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:38.699474 20526 heartbeater.cc:503] Master 127.5.198.254:33223 was elected leader, sending a full tablet report...
I20260929 01:10:38.700845 20805 tablet_bootstrap.cc:492] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9: No bootstrap required, opened a new log
I20260929 01:10:38.701155 20805 ts_tablet_manager.cc:1397] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260929 01:10:38.707206 20807 tablet_bootstrap.cc:492] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983: No bootstrap required, opened a new log
I20260929 01:10:38.707510 20807 ts_tablet_manager.cc:1397] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260929 01:10:38.707937 20806 tablet_bootstrap.cc:492] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97: No bootstrap required, opened a new log
I20260929 01:10:38.708269 20806 ts_tablet_manager.cc:1397] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260929 01:10:38.711478 20805 raft_consensus.cc:348] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } }
I20260929 01:10:38.711944 20805 raft_consensus.cc:374] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:10:38.712039 20805 raft_consensus.cc:729] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b173f6af4bc48168d80a0b064e3b7b9, State: Initialized, Role: FOLLOWER
I20260929 01:10:38.712710 20805 consensus_queue.cc:260] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } }
I20260929 01:10:38.714479 20805 ts_tablet_manager.cc:1428] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 01:10:38.715813 20807 raft_consensus.cc:348] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } }
I20260929 01:10:38.716224 20807 raft_consensus.cc:374] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:10:38.716336 20807 raft_consensus.cc:729] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02510081e38460ca54fbef2a3dd3983, State: Initialized, Role: FOLLOWER
I20260929 01:10:38.716954 20807 consensus_queue.cc:260] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } }
I20260929 01:10:38.718006 20658 heartbeater.cc:503] Master 127.5.198.254:33223 was elected leader, sending a full tablet report...
I20260929 01:10:38.718986 20806 raft_consensus.cc:348] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } }
I20260929 01:10:38.719290 20806 raft_consensus.cc:374] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:10:38.719347 20806 raft_consensus.cc:729] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a057a9edb6460ea39722b23a8cae97, State: Initialized, Role: FOLLOWER
I20260929 01:10:38.719820 20806 consensus_queue.cc:260] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } }
I20260929 01:10:38.719995 20807 ts_tablet_manager.cc:1428] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 01:10:38.720857 20789 heartbeater.cc:503] Master 127.5.198.254:33223 was elected leader, sending a full tablet report...
I20260929 01:10:38.721346 20806 ts_tablet_manager.cc:1428] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260929 01:10:38.734355 20813 raft_consensus.cc:482] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:10:38.734652 20813 raft_consensus.cc:504] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } }
I20260929 01:10:38.736372 20813 leader_election.cc:288] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b173f6af4bc48168d80a0b064e3b7b9 (127.5.198.193:32943), f02510081e38460ca54fbef2a3dd3983 (127.5.198.194:34065)
I20260929 01:10:38.743362 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.736824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:44657 (local address 127.5.198.194:34065)
0929 01:10:38.737029 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:38.737038 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:38.737064 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:10:38.737279 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:38.737284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:38.737362 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:38.737453 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:10:38.737880 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.738882 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.741427 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.742563 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.742724 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.742803 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:38.742996 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 01:10:38.743132 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:10:38.743409 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.736691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:34065 (local address 127.5.198.195:44657)
0929 01:10:38.736946 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:38.736977 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:38.737108 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:38.737497 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:38.737504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:38.737531 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:10:38.737730 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:38.737738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.739061 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.739065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:38.741279 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:38.741294 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.742718 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.742722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:38.742846 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.742871 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 01:10:38.742942 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 01:10:38.743105 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":37}
I20260929 01:10:38.743880 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.736709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:43247 (local address 127.5.198.193:32943)
0929 01:10:38.736906 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:38.736912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:38.737542 (+ 630us) server_negotiation.cc:407] Connection header received
0929 01:10:38.737674 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:38.737677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:38.737732 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:38.737824 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:10:38.738291 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.739230 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.741734 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.742914 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.743139 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.743205 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:38.743573 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 01:10:38.743720 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:10:38.744447 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.736731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:32943 (local address 127.5.198.195:43247)
0929 01:10:38.737398 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:38.737410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:38.737552 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:38.737843 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:38.737850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:38.737873 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:10:38.738146 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:38.738153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.739371 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.739376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:38.741591 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:38.741600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.743174 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.743180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:38.743350 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.743381 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 01:10:38.744068 (+ 687us) client_negotiation.cc:240] Negotiation successful
0929 01:10:38.744188 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":30,"thread_start_us":386,"threads_started":1}
I20260929 01:10:38.744349 20613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d419cc058406429ab88e5502e61cbc84" candidate_uuid: "05a057a9edb6460ea39722b23a8cae97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02510081e38460ca54fbef2a3dd3983" is_pre_election: true
I20260929 01:10:38.744911 20613 raft_consensus.cc:2454] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05a057a9edb6460ea39722b23a8cae97 in term 0.
I20260929 01:10:38.745494 20481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d419cc058406429ab88e5502e61cbc84" candidate_uuid: "05a057a9edb6460ea39722b23a8cae97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" is_pre_election: true
I20260929 01:10:38.745591 20680 leader_election.cc:302] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a057a9edb6460ea39722b23a8cae97, f02510081e38460ca54fbef2a3dd3983; no voters:
I20260929 01:10:38.745966 20813 raft_consensus.cc:2781] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:10:38.745971 20481 raft_consensus.cc:2454] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05a057a9edb6460ea39722b23a8cae97 in term 0.
I20260929 01:10:38.746131 20813 raft_consensus.cc:482] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:10:38.746237 20813 raft_consensus.cc:3037] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:10:38.748587 20813 raft_consensus.cc:504] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } }
I20260929 01:10:38.749215 20813 leader_election.cc:288] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [CANDIDATE]: Term 1 election: Requested vote from peers 0b173f6af4bc48168d80a0b064e3b7b9 (127.5.198.193:32943), f02510081e38460ca54fbef2a3dd3983 (127.5.198.194:34065)
I20260929 01:10:38.749635 20481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d419cc058406429ab88e5502e61cbc84" candidate_uuid: "05a057a9edb6460ea39722b23a8cae97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b173f6af4bc48168d80a0b064e3b7b9"
I20260929 01:10:38.749792 20613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d419cc058406429ab88e5502e61cbc84" candidate_uuid: "05a057a9edb6460ea39722b23a8cae97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02510081e38460ca54fbef2a3dd3983"
I20260929 01:10:38.749835 20481 raft_consensus.cc:3037] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:10:38.750051 20613 raft_consensus.cc:3037] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:10:38.753019 20481 raft_consensus.cc:2454] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05a057a9edb6460ea39722b23a8cae97 in term 1.
I20260929 01:10:38.753185 20613 raft_consensus.cc:2454] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05a057a9edb6460ea39722b23a8cae97 in term 1.
I20260929 01:10:38.753479 20678 leader_election.cc:302] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a057a9edb6460ea39722b23a8cae97, 0b173f6af4bc48168d80a0b064e3b7b9; no voters:
I20260929 01:10:38.753809 20813 raft_consensus.cc:2781] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:10:38.754273 20813 raft_consensus.cc:686] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [term 1 LEADER]: Becoming Leader. State: Replica: 05a057a9edb6460ea39722b23a8cae97, State: Running, Role: LEADER
I20260929 01:10:38.754849 20813 consensus_queue.cc:237] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } }
I20260929 01:10:38.759299 20339 catalog_manager.cc:5495] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05a057a9edb6460ea39722b23a8cae97 (127.5.198.195). New cstate: current_term: 1 leader_uuid: "05a057a9edb6460ea39722b23a8cae97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } health_report { overall_health: UNKNOWN } } }
I20260929 01:10:38.790552 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.785386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.5.198.195:46321)
0929 01:10:38.785594 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:38.785601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:38.785619 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:10:38.785717 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:38.785721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:38.785798 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:38.785957 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:10:38.786290 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.787029 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.788448 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.788885 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.789047 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.789514 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:10:38.790005 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:10:38.790157 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:38.790217 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 01:10:38.790293 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 01:10:38.798600 20481 raft_consensus.cc:1261] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Refusing update from remote peer 05a057a9edb6460ea39722b23a8cae97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:10:38.798609 20613 raft_consensus.cc:1261] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983 [term 1 FOLLOWER]: Refusing update from remote peer 05a057a9edb6460ea39722b23a8cae97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:10:38.799546 20815 consensus_queue.cc:1035] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:10:38.799911 20813 consensus_queue.cc:1035] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:10:38.826229 20339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:10:38.826488 20339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:10:38.830288 20339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023: Sending DeleteTablet for 3 replicas of tablet d419cc058406429ab88e5502e61cbc84
I20260929 01:10:38.831485 20724 tablet_service.cc:1514] Processing DeleteTablet for tablet d419cc058406429ab88e5502e61cbc84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:10:38 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:50594
I20260929 01:10:38.831533 20593 tablet_service.cc:1514] Processing DeleteTablet for tablet d419cc058406429ab88e5502e61cbc84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:10:38 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:33036
I20260929 01:10:38.831705 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 20377
I20260929 01:10:38.832286 20461 tablet_service.cc:1514] Processing DeleteTablet for tablet d419cc058406429ab88e5502e61cbc84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:10:38 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:57854
I20260929 01:10:38.839622 20829 tablet_replica.cc:330] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9: stopping tablet replica
I20260929 01:10:38.840209 20829 raft_consensus.cc:2229] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:10:38.840889 20829 raft_consensus.cc:2258] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:10:38.845934 20828 tablet_replica.cc:330] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983: stopping tablet replica
I20260929 01:10:38.846560 20828 raft_consensus.cc:2229] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:10:38.847153 20828 raft_consensus.cc:2258] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:10:38.849707 20828 ts_tablet_manager.cc:1905] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:10:38.854395 20829 ts_tablet_manager.cc:1905] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:10:38.855224 20827 tablet_replica.cc:330] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97: stopping tablet replica
I20260929 01:10:38.855810 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.833423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57860 (local address 127.5.198.193:32943)
0929 01:10:38.833669 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:38.833678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:38.833701 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:10:38.833773 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:38.833779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:38.833871 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:38.833986 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:38.836418 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.837990 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.840716 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.841710 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.841857 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.846591 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:38.846614 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:38.846639 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:38.846693 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:38.851512 (+ 4819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:38.852713 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:38.852725 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:38.852731 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:38.852913 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:38.853486 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:38.853494 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:38.853500 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:38.854120 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:38.854539 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:38.855145 (+ 606us) server_negotiation.cc:299] Negotiation successful
0929 01:10:38.855387 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:10:38.855743 20827 raft_consensus.cc:2229] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:10:38.856580 20521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:38.856878 20827 raft_consensus.cc:2258] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:10:38.859664 20827 ts_tablet_manager.cc:1905] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:10:38.860236 20828 ts_tablet_manager.cc:1918] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:10:38.860519 20828 log.cc:1196] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/d419cc058406429ab88e5502e61cbc84
I20260929 01:10:38.861172 20828 ts_tablet_manager.cc:1939] T d419cc058406429ab88e5502e61cbc84 P f02510081e38460ca54fbef2a3dd3983: Deleting consensus metadata
I20260929 01:10:38.863463 20327 catalog_manager.cc:4841] TS f02510081e38460ca54fbef2a3dd3983 (127.5.198.194:34065): tablet d419cc058406429ab88e5502e61cbc84 (table test-table [id=9891632cc2994ce98edf3fc110005c27]) successfully deleted
I20260929 01:10:38.868376 20827 ts_tablet_manager.cc:1918] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:10:38.868587 20827 log.cc:1196] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/d419cc058406429ab88e5502e61cbc84
I20260929 01:10:38.869208 20827 ts_tablet_manager.cc:1939] T d419cc058406429ab88e5502e61cbc84 P 05a057a9edb6460ea39722b23a8cae97: Deleting consensus metadata
I20260929 01:10:38.871205 20327 catalog_manager.cc:4841] TS 05a057a9edb6460ea39722b23a8cae97 (127.5.198.195:46321): tablet d419cc058406429ab88e5502e61cbc84 (table test-table [id=9891632cc2994ce98edf3fc110005c27]) successfully deleted
I20260929 01:10:38.954089 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 20377
W20260929 01:10:38.978880 20325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.198.193:32943 (error 108)
I20260929 01:10:38.979260 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 20530
I20260929 01:10:38.986202 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.985644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:32943 (local address 127.0.0.1:57864)
0929 01:10:38.985852 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:38.985990 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 01:10:38.986598 20325 catalog_manager.cc:4584] TS 0b173f6af4bc48168d80a0b064e3b7b9 (127.5.198.193:32943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d419cc058406429ab88e5502e61cbc84: Network error: Client connection negotiation failed: client connection to 127.5.198.193:32943: connect: Connection refused (error 111)
I20260929 01:10:38.996745 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:38.980370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33040 (local address 127.5.198.194:34065)
0929 01:10:38.981200 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:38.981210 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:38.981238 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:10:38.981330 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:38.981339 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:38.981468 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:38.981610 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:38.983410 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.984946 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.986613 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:38.987253 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:38.987384 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:38.990223 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:38.990240 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:38.990259 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:38.990300 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:38.993990 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:38.994649 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:38.994658 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:38.994662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:38.994806 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:38.995137 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:38.995142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:38.995146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:38.995596 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:38.995916 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:38.996118 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 01:10:38.996339 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643}
I20260929 01:10:38.997537 20653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:39.040553 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:39.040016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:32943 (local address 127.0.0.1:57880)
0929 01:10:39.040237 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:39.040348 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 01:10:39.041741 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:39.041292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:32943 (local address 127.0.0.1:57892)
0929 01:10:39.041471 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:39.041579 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 01:10:39.087538 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 20530
I20260929 01:10:39.102407 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:39.101810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:32943 (local address 127.0.0.1:57896)
0929 01:10:39.102074 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:39.102197 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 01:10:39.103616 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:39.103184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:32943 (local address 127.0.0.1:57902)
0929 01:10:39.103349 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:39.103444 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 01:10:39.105443 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 20661
I20260929 01:10:39.121706 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:39.106169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.5.198.195:46321)
0929 01:10:39.106527 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:39.106531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:39.106548 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:10:39.106599 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:39.106603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:39.106675 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:39.106760 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:39.108745 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:39.109606 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:39.111430 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:39.111842 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:39.111956 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:39.115521 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:39.115535 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:39.115556 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:39.115594 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:39.118694 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:39.119560 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:39.119566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:39.119568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:39.119738 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:39.120164 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:39.120167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:39.120169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:39.120582 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:39.120849 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:39.121193 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 01:10:39.121355 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260929 01:10:39.122359 20784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:39.184319 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:39.183855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:32943 (local address 127.0.0.1:57908)
0929 01:10:39.184066 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:39.184160 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 01:10:39.185417 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:39.185008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:32943 (local address 127.0.0.1:57916)
0929 01:10:39.185220 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:39.185301 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260929 01:10:39.243355 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 20661
I20260929 01:10:39.266505 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 20307
I20260929 01:10:39.267797 20370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:39.382845 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:39.382233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:32943 (local address 127.0.0.1:57930)
0929 01:10:39.382470 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:39.382578 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 01:10:39.383033 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 20307
I20260929 01:10:39.405221 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:33223
--webserver_interface=127.5.198.254
--webserver_port=45499
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:33223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:10:39.736368 20835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:39.736692 20835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:39.736784 20835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:39.744108 20835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:10:39.744221 20835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:39.744269 20835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:39.744318 20835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:10:39.744364 20835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:10:39.754266 20835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:33223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:33223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:39.755527 20835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:39.756801 20835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:39.769260 20841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:39.769279 20840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:39.769287 20843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:39.770377 20835 server_base.cc:1034] running on GCE node
I20260929 01:10:39.771188 20835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:39.772711 20835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:39.773962 20835 hybrid_clock.cc:648] HybridClock initialized: now 1790644239773894 us; error 73 us; skew 500 ppm
I20260929 01:10:39.774339 20835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:39 dist-test-slave-41cl krb5kdc[19756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644239, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:39.777581 20835 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:10:39.779249 20835 webserver.cc:460] Webserver started at http://127.5.198.254:45499/ using document root <none> and password file <none>
I20260929 01:10:39.779951 20835 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:39.780113 20835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:39.785679 20835 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 01:10:39.788671 20850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:39.790236 20835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:10:39.790438 20835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "d352ec48235d4fdab189332b74e40023"
format_stamp: "Formatted at 2026-09-29 01:10:33 on dist-test-slave-41cl"
server_key: "0ebafa4debdbe547f3ce4fc73bbf82c5"
server_key_iv: "c86f8a7444c260873cbcb28bc31f433a"
server_key_version: "encryptionkey@0"
I20260929 01:10:39.791512 20835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:39.828714 20835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:39.833014 20835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:39.833292 20835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:39.853202 20835 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:33223
I20260929 01:10:39.853221 20901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:33223 every 8 connection(s)
I20260929 01:10:39.854751 20835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:10:39.858238 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 20835
I20260929 01:10:39.859413 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:32943
--local_ip_for_outbound_sockets=127.5.198.193
--tserver_master_addrs=127.5.198.254:33223
--webserver_port=46515
--webserver_interface=127.5.198.193
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:10:39.865160 20902 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:10:39.870123 20902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023: Bootstrap starting.
I20260929 01:10:39.892665 20902 log.cc:824] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:39.909667 20902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:10:39.910252 20902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023: Bootstrap complete.
I20260929 01:10:39.919374 20902 raft_consensus.cc:348] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } }
I20260929 01:10:39.920233 20902 raft_consensus.cc:729] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d352ec48235d4fdab189332b74e40023, State: Initialized, Role: FOLLOWER
I20260929 01:10:39.920918 20902 consensus_queue.cc:260] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } }
I20260929 01:10:39.921147 20902 raft_consensus.cc:388] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:10:39.921258 20902 raft_consensus.cc:482] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:10:39.921456 20902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:10:39.925297 20902 raft_consensus.cc:504] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } }
I20260929 01:10:39.925798 20902 leader_election.cc:302] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d352ec48235d4fdab189332b74e40023; no voters:
I20260929 01:10:39.926497 20902 leader_election.cc:288] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 01:10:39.926647 20906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:10:39.927685 20906 raft_consensus.cc:686] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [term 3 LEADER]: Becoming Leader. State: Replica: d352ec48235d4fdab189332b74e40023, State: Running, Role: LEADER
I20260929 01:10:39.928236 20906 consensus_queue.cc:237] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } }
I20260929 01:10:39.928857 20902 sys_catalog.cc:564] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:10:39.931030 20907 sys_catalog.cc:455] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d352ec48235d4fdab189332b74e40023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } } }
I20260929 01:10:39.931324 20907 sys_catalog.cc:458] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:39.930670 20908 sys_catalog.cc:455] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d352ec48235d4fdab189332b74e40023. Latest consensus state: current_term: 3 leader_uuid: "d352ec48235d4fdab189332b74e40023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352ec48235d4fdab189332b74e40023" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33223 } } }
I20260929 01:10:39.931905 20908 sys_catalog.cc:458] T 00000000000000000000000000000000 P d352ec48235d4fdab189332b74e40023 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:39.935593 20911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:10:39.948771 20911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:10:39.951407 20911 catalog_manager.cc:1259] Loaded cluster ID: 527a6028225e47d1835e808cb5fcb580
I20260929 01:10:39.951556 20911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 01:10:39.954195 20911 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd7fbe419ad google::LogMessage::Fail() at ??:0
@ 0x7fd7fbe4488a google::LogMessage::SendToLog() at ??:0
@ 0x7fd7fbe414de google::LogMessage::Flush() at ??:0
@ 0x7fd7fbe42a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd811874512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd811871976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd8118f9610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd8118f9439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd7fea8d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd7fea9091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd7fea90799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd7fea6beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd801d456db start_thread at ??:0
@ 0x7fd7facdb61f clone at ??:0
W20260929 01:10:40.267592 20904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:40.267880 20904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:40.267987 20904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:40.275537 20904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:40.275650 20904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:40.275763 20904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:10:40.285712 20904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:32943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=46515
--tserver_master_addrs=127.5.198.254:33223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:40.286989 20904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:40.288266 20904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:40.301549 20943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:40.301626 20944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:40.301582 20946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:40.306109 20904 server_base.cc:1034] running on GCE node
I20260929 01:10:40.306710 20904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:40.307698 20904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:40.308903 20904 hybrid_clock.cc:648] HybridClock initialized: now 1790644240308836 us; error 54 us; skew 500 ppm
I20260929 01:10:40.309274 20904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:40 dist-test-slave-41cl krb5kdc[19756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644240, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:40.312557 20904 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:10:40.314073 20904 webserver.cc:460] Webserver started at http://127.5.198.193:46515/ using document root <none> and password file <none>
I20260929 01:10:40.314775 20904 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:40.314877 20904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:40.320338 20904 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:10:40.325274 20953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:40.326813 20904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:10:40.326951 20904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "0b173f6af4bc48168d80a0b064e3b7b9"
format_stamp: "Formatted at 2026-09-29 01:10:34 on dist-test-slave-41cl"
server_key: "ccb0e4e25b83f8362066dc7358e362ad"
server_key_iv: "86e1a775a1dfbbdc3c801956561ac8f8"
server_key_version: "encryptionkey@0"
I20260929 01:10:40.327888 20904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:40.350593 20904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:40.354796 20904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:40.355064 20904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:40.355995 20904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:40.357841 20960 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 01:10:40.363329 20904 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 01:10:40.363472 20904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 01:10:40.363633 20904 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 01:10:40.366731 20904 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 01:10:40.366838 20904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260929 01:10:40.367216 20960 tablet_bootstrap.cc:492] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9: Bootstrap starting.
I20260929 01:10:40.375756 20960 log.cc:824] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:40.396169 20960 tablet_bootstrap.cc:492] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:10:40.397157 20960 tablet_bootstrap.cc:492] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9: Bootstrap complete.
I20260929 01:10:40.397722 20960 ts_tablet_manager.cc:1397] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.008s
I20260929 01:10:40.407400 20960 raft_consensus.cc:348] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } }
I20260929 01:10:40.409004 20960 raft_consensus.cc:729] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b173f6af4bc48168d80a0b064e3b7b9, State: Initialized, Role: FOLLOWER
I20260929 01:10:40.409688 20960 consensus_queue.cc:260] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } }
I20260929 01:10:40.411281 20960 ts_tablet_manager.cc:1428] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9: Time spent starting tablet: real 0.013s user 0.005s sys 0.009s
I20260929 01:10:40.424093 20904 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:32943
I20260929 01:10:40.424170 21068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:32943 every 8 connection(s)
I20260929 01:10:40.425446 20904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:10:40.429355 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 20904
I20260929 01:10:40.430013 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:34065
--local_ip_for_outbound_sockets=127.5.198.194
--tserver_master_addrs=127.5.198.254:33223
--webserver_port=45875
--webserver_interface=127.5.198.194
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:10:40.430238 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:40.428246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:45847)
0929 01:10:40.429054 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:40.429272 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":386,"threads_started":1}
W20260929 01:10:40.430868 21069 heartbeater.cc:644] Failed to heartbeat to 127.5.198.254:33223 (0 consecutive failures): Network error: Failed to ping master at 127.5.198.254:33223: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
I20260929 01:10:40.431782 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:40.431356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:35767)
0929 01:10:40.431591 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:40.431665 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260929 01:10:40.432691 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:40.432283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:53373)
0929 01:10:40.432437 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:40.432563 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 01:10:40.432936 21069 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:10:40.763186 21072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:40.763484 21072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:40.763672 21072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:40.771488 21072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:40.771606 21072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:40.771793 21072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:10:40.782333 21072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:34065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=45875
--tserver_master_addrs=127.5.198.254:33223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:40.783607 21072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:40.785001 21072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:40.798530 21080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:40.798596 21077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:40.798573 21078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:40.798815 21072 server_base.cc:1034] running on GCE node
I20260929 01:10:40.799659 21072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:40.800626 21072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:40.801824 21072 hybrid_clock.cc:648] HybridClock initialized: now 1790644240801759 us; error 53 us; skew 500 ppm
I20260929 01:10:40.802170 21072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:40 dist-test-slave-41cl krb5kdc[19756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644240, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:40.805222 21072 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:10:40.806591 21072 webserver.cc:460] Webserver started at http://127.5.198.194:45875/ using document root <none> and password file <none>
I20260929 01:10:40.807209 21072 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:40.807301 21072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:40.812618 21072 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:10:40.815276 21087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:40.816740 21072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:10:40.816927 21072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "f02510081e38460ca54fbef2a3dd3983"
format_stamp: "Formatted at 2026-09-29 01:10:35 on dist-test-slave-41cl"
server_key: "650f214e68c026ad44a28a70351c9bf6"
server_key_iv: "050f9944557659bf6e1e7b10d11ba925"
server_key_version: "encryptionkey@0"
I20260929 01:10:40.817963 21072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:40.839646 21072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:40.843911 21072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:40.844193 21072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:40.845239 21072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:40.846552 21072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:10:40.846678 21072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:40.846814 21072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:10:40.846899 21072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:40.881021 21072 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:34065
I20260929 01:10:40.881074 21199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:34065 every 8 connection(s)
I20260929 01:10:40.882457 21072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:10:40.883486 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 21072
I20260929 01:10:40.884299 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:46321
--local_ip_for_outbound_sockets=127.5.198.195
--tserver_master_addrs=127.5.198.254:33223
--webserver_port=39191
--webserver_interface=127.5.198.195
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:10:40.888271 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:40.885994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:34747)
0929 01:10:40.886830 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:40.887129 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":308,"threads_started":1}
W20260929 01:10:40.889037 21200 heartbeater.cc:644] Failed to heartbeat to 127.5.198.254:33223 (0 consecutive failures): Network error: Failed to ping master at 127.5.198.254:33223: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
I20260929 01:10:40.889874 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:40.889532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:44529)
0929 01:10:40.889692 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:40.889770 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 01:10:40.890782 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:40.890447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:52335)
0929 01:10:40.890590 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:40.890673 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 01:10:40.891072 21200 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:10:41.208173 21203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:41.208432 21203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:41.208650 21203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:41.216058 21203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:41.216166 21203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:41.216395 21203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:10:41.226312 21203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:46321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=39191
--tserver_master_addrs=127.5.198.254:33223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:41.227600 21203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:41.228971 21203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:41.242408 21209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:41.242313 21208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:41.242440 21211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:41.243463 21203 server_base.cc:1034] running on GCE node
I20260929 01:10:41.244066 21203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:41.245057 21203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:41.246286 21203 hybrid_clock.cc:648] HybridClock initialized: now 1790644241246237 us; error 43 us; skew 500 ppm
I20260929 01:10:41.246652 21203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:41 dist-test-slave-41cl krb5kdc[19756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644241, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:41.249809 21203 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:10:41.251251 21203 webserver.cc:460] Webserver started at http://127.5.198.195:39191/ using document root <none> and password file <none>
I20260929 01:10:41.251921 21203 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:41.252063 21203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:41.257294 21203 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:10:41.259923 21218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:41.261281 21203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:10:41.261462 21203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "05a057a9edb6460ea39722b23a8cae97"
format_stamp: "Formatted at 2026-09-29 01:10:35 on dist-test-slave-41cl"
server_key: "5d01832eccae62fd211499228a73868d"
server_key_iv: "dcdd18ab5da7d35156ea2b37b01fa6bd"
server_key_version: "encryptionkey@0"
I20260929 01:10:41.262425 21203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:41.282140 21203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:41.285995 21203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:41.286263 21203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:41.287238 21203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:41.288587 21203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:10:41.288715 21203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:41.288853 21203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:10:41.288944 21203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:41.323159 21203 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:46321
I20260929 01:10:41.323243 21330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:46321 every 8 connection(s)
I20260929 01:10:41.324661 21203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:10:41.328284 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 21203
I20260929 01:10:41.329735 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:41.327491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:60167)
0929 01:10:41.328362 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:41.328708 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":335,"threads_started":1}
W20260929 01:10:41.330485 21331 heartbeater.cc:644] Failed to heartbeat to 127.5.198.254:33223 (0 consecutive failures): Network error: Failed to ping master at 127.5.198.254:33223: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
I20260929 01:10:41.331219 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:41.330899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:33929)
0929 01:10:41.331057 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:41.331118 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 01:10:41.332029 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:41.331755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:32961)
0929 01:10:41.331864 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:41.331930 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 01:10:41.332240 21331 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 01:10:41.434266 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:41.433511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:59003)
0929 01:10:41.433963 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:41.434057 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":210,"threads_started":1}
I20260929 01:10:41.892457 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:41.891629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:52351)
0929 01:10:41.892119 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:41.892214 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":240,"threads_started":1}
I20260929 01:10:42.184916 21337 raft_consensus.cc:482] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:10:42.185178 21337 raft_consensus.cc:504] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } }
I20260929 01:10:42.186749 21337 leader_election.cc:288] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05a057a9edb6460ea39722b23a8cae97 (127.5.198.195:46321), f02510081e38460ca54fbef2a3dd3983 (127.5.198.194:34065)
Sep 29 01:10:42 dist-test-slave-41cl krb5kdc[19756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644240, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
Sep 29 01:10:42 dist-test-slave-41cl krb5kdc[19756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644240, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:10:42.218326 21341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:42.188107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:43521 (local address 127.5.198.194:34065)
0929 01:10:42.189039 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:42.189057 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:42.189444 (+ 387us) server_negotiation.cc:407] Connection header received
0929 01:10:42.191194 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:42.191203 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:42.191548 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:42.191869 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:42.194059 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:42.195525 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:42.197063 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:42.197825 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:42.198025 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:42.209121 (+ 11096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:42.209132 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:42.209150 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:42.209207 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:42.213273 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:42.214506 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:42.214515 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:42.214523 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:42.214724 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:42.215360 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:42.215366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:42.215369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:42.216084 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:42.216276 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:42.217475 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0929 01:10:42.217918 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":580,"threads_started":1}
I20260929 01:10:42.218366 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:42.187217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:43955 (local address 127.5.198.195:46321)
0929 01:10:42.189464 (+ 2247us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:42.189477 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:42.189514 (+ 37us) server_negotiation.cc:407] Connection header received
0929 01:10:42.191106 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:42.191114 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:42.191535 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:42.191853 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:42.193945 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:42.195148 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:42.197021 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:42.198344 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:42.198550 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:42.208939 (+ 10389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:42.208953 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:42.208977 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:42.209039 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:42.213273 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:42.214506 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:42.214515 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:42.214523 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:42.214724 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:42.215360 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:42.215365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:42.215369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:42.216084 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:42.216276 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:42.217471 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0929 01:10:42.217907 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2069,"thread_start_us":1955,"threads_started":1}
I20260929 01:10:42.218346 21340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:42.187763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:34065 (local address 127.5.198.193:43521)
0929 01:10:42.189312 (+ 1549us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:42.189331 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:42.189480 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:42.192004 (+ 2524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:42.192017 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:42.193072 (+ 1055us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:42.193849 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:42.193866 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:42.195702 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:42.195706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:42.196888 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:42.196900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:42.198027 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:42.198034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:42.198301 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:42.205768 (+ 7467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:42.205808 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:42.208791 (+ 2983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:42.213464 (+ 4673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:42.213475 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:42.213493 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:42.214312 (+ 819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:42.214851 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:42.214857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:42.214861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:42.215218 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:42.216252 (+ 1034us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:42.216262 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:42.216842 (+ 580us) client_negotiation.cc:769] Sending connection context
0929 01:10:42.217168 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 01:10:42.217844 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1428,"mutex_wait_us":257,"spinlock_wait_cycles":70400,"thread_start_us":1221,"threads_started":1}
I20260929 01:10:42.218369 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:42.186907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:46321 (local address 127.5.198.193:43955)
0929 01:10:42.187482 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:42.187510 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:42.188127 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:42.192008 (+ 3881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:42.192017 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:42.193074 (+ 1057us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:42.193711 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:42.193732 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:42.195319 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:42.195325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:42.196882 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:42.196894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:42.198563 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:42.198570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:42.198757 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:42.205762 (+ 7005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:42.205794 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:42.208567 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:42.213464 (+ 4897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:42.213475 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:42.213493 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:42.214312 (+ 819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:42.214851 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:42.214857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:42.214861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:42.215218 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:42.216252 (+ 1034us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:42.216262 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:42.216842 (+ 580us) client_negotiation.cc:769] Sending connection context
0929 01:10:42.217168 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 01:10:42.217739 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":233,"spinlock_wait_cycles":23552,"thread_start_us":373,"threads_started":1}
W20260929 01:10:42.222231 20957 leader_election.cc:341] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05a057a9edb6460ea39722b23a8cae97 (127.5.198.195:46321): Not found: Tablet not found: d419cc058406429ab88e5502e61cbc84
W20260929 01:10:42.223201 20957 leader_election.cc:341] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f02510081e38460ca54fbef2a3dd3983 (127.5.198.194:34065): Not found: Tablet not found: d419cc058406429ab88e5502e61cbc84
I20260929 01:10:42.223323 20957 leader_election.cc:302] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b173f6af4bc48168d80a0b064e3b7b9; no voters: 05a057a9edb6460ea39722b23a8cae97, f02510081e38460ca54fbef2a3dd3983
I20260929 01:10:42.223690 21337 raft_consensus.cc:2726] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:10:42.333724 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:42.332905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:52537)
0929 01:10:42.333365 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:42.333503 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":239,"threads_started":1}
I20260929 01:10:42.435667 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:42.435236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:59999)
0929 01:10:42.435432 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:42.435527 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 01:10:42.894089 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:42.893444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:50229)
0929 01:10:42.893852 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:42.893938 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":218,"threads_started":1}
I20260929 01:10:43.335230 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:43.334609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:52137)
0929 01:10:43.335007 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:43.335085 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":212,"threads_started":1}
I20260929 01:10:43.437398 21345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:43.436662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:53927)
0929 01:10:43.437136 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:43.437222 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":249,"threads_started":1}
I20260929 01:10:43.895668 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:43.894950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:33677)
0929 01:10:43.895385 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:43.895491 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":236,"threads_started":1}
I20260929 01:10:44.003037 21347 raft_consensus.cc:482] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:10:44.003266 21347 raft_consensus.cc:504] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } }
I20260929 01:10:44.004055 21347 leader_election.cc:288] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05a057a9edb6460ea39722b23a8cae97 (127.5.198.195:46321), f02510081e38460ca54fbef2a3dd3983 (127.5.198.194:34065)
W20260929 01:10:44.005182 20957 leader_election.cc:341] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05a057a9edb6460ea39722b23a8cae97 (127.5.198.195:46321): Not found: Tablet not found: d419cc058406429ab88e5502e61cbc84
W20260929 01:10:44.005398 20957 leader_election.cc:341] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f02510081e38460ca54fbef2a3dd3983 (127.5.198.194:34065): Not found: Tablet not found: d419cc058406429ab88e5502e61cbc84
I20260929 01:10:44.005484 20957 leader_election.cc:302] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b173f6af4bc48168d80a0b064e3b7b9; no voters: 05a057a9edb6460ea39722b23a8cae97, f02510081e38460ca54fbef2a3dd3983
I20260929 01:10:44.005764 21347 raft_consensus.cc:2726] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:10:44.336714 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:44.336034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:55389)
0929 01:10:44.336450 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:44.336558 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":221,"threads_started":1}
I20260929 01:10:44.439311 21349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:44.438393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:58889)
0929 01:10:44.438873 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:44.439113 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":241,"threads_started":1}
I20260929 01:10:44.897506 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:44.896744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:54785)
0929 01:10:44.897242 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:44.897325 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":294,"threads_started":1}
I20260929 01:10:45.338265 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:45.337527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:34819)
0929 01:10:45.338005 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:45.338089 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":222,"threads_started":1}
I20260929 01:10:45.441181 21352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:45.440297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:52765)
0929 01:10:45.440867 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:45.440943 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":279,"threads_started":1}
I20260929 01:10:45.899192 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:45.898412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:38439)
0929 01:10:45.898915 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:45.898992 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":264,"threads_started":1}
I20260929 01:10:46.339907 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:46.339220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:51143)
0929 01:10:46.339664 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:46.339743 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":250,"threads_started":1}
I20260929 01:10:46.442723 21355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:46.442059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:43643)
0929 01:10:46.442472 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:46.442565 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":218,"threads_started":1}
I20260929 01:10:46.842269 21356 raft_consensus.cc:482] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:10:46.842469 21356 raft_consensus.cc:504] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } }
I20260929 01:10:46.843086 21356 leader_election.cc:288] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05a057a9edb6460ea39722b23a8cae97 (127.5.198.195:46321), f02510081e38460ca54fbef2a3dd3983 (127.5.198.194:34065)
W20260929 01:10:46.844163 20957 leader_election.cc:341] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05a057a9edb6460ea39722b23a8cae97 (127.5.198.195:46321): Not found: Tablet not found: d419cc058406429ab88e5502e61cbc84
W20260929 01:10:46.844406 20957 leader_election.cc:341] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f02510081e38460ca54fbef2a3dd3983 (127.5.198.194:34065): Not found: Tablet not found: d419cc058406429ab88e5502e61cbc84
I20260929 01:10:46.844553 20957 leader_election.cc:302] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b173f6af4bc48168d80a0b064e3b7b9; no voters: 05a057a9edb6460ea39722b23a8cae97, f02510081e38460ca54fbef2a3dd3983
I20260929 01:10:46.844859 21356 raft_consensus.cc:2726] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:10:46.901022 21357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:46.900062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:59087)
0929 01:10:46.900685 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:46.900792 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":290,"threads_started":1}
I20260929 01:10:47.341553 21358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:47.340825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:53143)
0929 01:10:47.341285 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:47.341364 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":260,"threads_started":1}
I20260929 01:10:47.444481 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:47.443707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:47061)
0929 01:10:47.444160 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:47.444252 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":237,"threads_started":1}
I20260929 01:10:47.903007 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:47.902136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:57661)
0929 01:10:47.902717 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:47.902810 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":266,"threads_started":1}
I20260929 01:10:48.343350 21361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:48.342531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:55725)
0929 01:10:48.343088 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:48.343169 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":226,"threads_started":1}
I20260929 01:10:48.446277 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:48.445485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:48645)
0929 01:10:48.446034 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:48.446115 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":270,"threads_started":1}
I20260929 01:10:48.904639 21363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:48.903952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:39497)
0929 01:10:48.904356 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:48.904445 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":208,"threads_started":1}
I20260929 01:10:49.345196 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:49.344397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:40887)
0929 01:10:49.344897 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:49.344987 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":239,"threads_started":1}
I20260929 01:10:49.448195 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:49.447305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:57415)
0929 01:10:49.447810 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:49.447993 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":286,"threads_started":1}
I20260929 01:10:49.906477 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:49.905628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:55169)
0929 01:10:49.906090 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:49.906223 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":206,"threads_started":1}
I20260929 01:10:50.347060 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:50.346186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:43323)
0929 01:10:50.346765 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:50.346855 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":313,"threads_started":1}
I20260929 01:10:50.450101 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:50.449340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:32953)
0929 01:10:50.449844 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:50.449937 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":260,"threads_started":1}
I20260929 01:10:50.908326 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:50.907568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:38329)
0929 01:10:50.908058 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:50.908145 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":249,"threads_started":1}
I20260929 01:10:51.348932 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:51.348075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:54839)
0929 01:10:51.348671 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:51.348753 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":225,"threads_started":1}
I20260929 01:10:51.367530 21371 raft_consensus.cc:482] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:10:51.367704 21371 raft_consensus.cc:504] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a057a9edb6460ea39722b23a8cae97" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46321 } } peers { permanent_uuid: "0b173f6af4bc48168d80a0b064e3b7b9" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 32943 } } peers { permanent_uuid: "f02510081e38460ca54fbef2a3dd3983" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 34065 } }
I20260929 01:10:51.368386 21371 leader_election.cc:288] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05a057a9edb6460ea39722b23a8cae97 (127.5.198.195:46321), f02510081e38460ca54fbef2a3dd3983 (127.5.198.194:34065)
W20260929 01:10:51.369407 20957 leader_election.cc:341] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05a057a9edb6460ea39722b23a8cae97 (127.5.198.195:46321): Not found: Tablet not found: d419cc058406429ab88e5502e61cbc84
W20260929 01:10:51.369704 20957 leader_election.cc:341] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f02510081e38460ca54fbef2a3dd3983 (127.5.198.194:34065): Not found: Tablet not found: d419cc058406429ab88e5502e61cbc84
I20260929 01:10:51.369819 20957 leader_election.cc:302] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b173f6af4bc48168d80a0b064e3b7b9; no voters: 05a057a9edb6460ea39722b23a8cae97, f02510081e38460ca54fbef2a3dd3983
I20260929 01:10:51.370101 21371 raft_consensus.cc:2726] T d419cc058406429ab88e5502e61cbc84 P 0b173f6af4bc48168d80a0b064e3b7b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:10:51.451884 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:51.451095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:34663)
0929 01:10:51.451589 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:51.451691 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":247,"threads_started":1}
I20260929 01:10:51.910372 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:51.909341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:44941)
0929 01:10:51.910040 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:51.910141 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":380,"threads_started":1}
I20260929 01:10:52.350694 21374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:52.349900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:34529)
0929 01:10:52.350427 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:52.350516 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":266,"threads_started":1}
I20260929 01:10:52.453707 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:52.452922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:58895)
0929 01:10:52.453446 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:52.453540 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":308,"threads_started":1}
I20260929 01:10:52.912125 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:52.911419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:40991)
0929 01:10:52.911867 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:52.911955 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":226,"threads_started":1}
I20260929 01:10:53.352768 21377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:53.351763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:50643)
0929 01:10:53.352299 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:53.352454 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":294,"threads_started":1}
I20260929 01:10:53.455559 21378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:53.454732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:46401)
0929 01:10:53.455270 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:53.455363 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":307,"threads_started":1}
I20260929 01:10:53.913975 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:53.913196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:53521)
0929 01:10:53.913711 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:53.913819 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":272,"threads_started":1}
I20260929 01:10:54.354549 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:54.353699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:49151)
0929 01:10:54.354233 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:54.354351 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":271,"threads_started":1}
I20260929 01:10:54.457334 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:54.456545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:57917)
0929 01:10:54.457047 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:54.457146 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":229,"threads_started":1}
I20260929 01:10:54.915839 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:54.915071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:60029)
0929 01:10:54.915544 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:54.915631 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":242,"threads_started":1}
I20260929 01:10:55.356352 21383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:55.355538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:57263)
0929 01:10:55.356035 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:55.356155 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":266,"threads_started":1}
I20260929 01:10:55.459789 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:55.458890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.193:41107)
0929 01:10:55.459499 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:55.459604 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":315,"threads_started":1}
I20260929 01:10:55.917990 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:55.917190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.194:40839)
0929 01:10:55.917687 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:55.917784 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":238,"threads_started":1}
I20260929 01:10:56.330817 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 20904
I20260929 01:10:56.353300 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:56.331922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36248 (local address 127.5.198.193:32943)
0929 01:10:56.332526 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:56.332536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:56.332567 (+ 31us) server_negotiation.cc:407] Connection header received
0929 01:10:56.332643 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:56.332654 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:56.332956 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:56.333153 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:56.334900 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:56.335897 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:56.337308 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:56.337841 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:56.337941 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:56.347429 (+ 9488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:56.347442 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:56.347456 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:56.347508 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:56.350437 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:56.351228 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:56.351235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:56.351237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:56.351371 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:56.351702 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:56.351705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:56.351707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:56.352076 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:56.352293 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:56.352709 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 01:10:56.352957 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":303,"threads_started":1}
I20260929 01:10:56.354640 21064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:56.358414 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:56.357552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33223 (local address 127.5.198.195:36165)
0929 01:10:56.358050 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:56.358195 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:33223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":251,"threads_started":1}
I20260929 01:10:56.444339 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 20904
I20260929 01:10:56.461623 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 21072
I20260929 01:10:56.475716 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:56.462411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.5.198.194:34065)
0929 01:10:56.462931 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:56.462935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:56.462951 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:10:56.463007 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:56.463011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:56.463101 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:56.463196 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:56.464619 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:56.465471 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:56.466853 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:56.467332 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:56.467443 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:56.470541 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:56.470547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:56.470550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:56.470583 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:56.473247 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:56.473941 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:56.473945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:56.473947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:56.474099 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:56.474430 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:56.474434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:56.474436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:56.474706 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:56.474903 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:56.475145 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 01:10:56.475313 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":320,"threads_started":1}
I20260929 01:10:56.476570 21195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:56.560439 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 21072
I20260929 01:10:56.577811 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 21203
I20260929 01:10:56.590991 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:56.578536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.5.198.195:46321)
0929 01:10:56.579119 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:56.579123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:56.579140 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:10:56.579196 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:56.579200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:56.579271 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:56.579362 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:56.580800 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:56.581572 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:56.582750 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:56.583167 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:56.583264 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:56.585825 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:56.585830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:56.585833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:56.585865 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:56.588408 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:56.589096 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:56.589099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:56.589102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:56.589248 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:56.589568 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:56.589571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:56.589573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:56.589826 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:56.590017 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:56.590375 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 01:10:56.590557 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":324,"threads_started":1}
I20260929 01:10:56.591714 21326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:10:56.677234 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 21203
2026-09-29T01:10:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24428 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:10:56 dist-test-slave-41cl krb5kdc[21395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:10:56 dist-test-slave-41cl krb5kdc[21395](info): set up 2 sockets
Sep 29 01:10:56 dist-test-slave-41cl krb5kdc[21395](info): commencing operation
krb5kdc: starting...
W20260929 01:10:57.781019 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:10:57 dist-test-slave-41cl krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:10:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:10:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:10:58.025179 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:38197
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:38197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:10:58.344456 21408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:58.344748 21408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:58.344908 21408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:58.352185 21408 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 01:10:58.352284 21408 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 01:10:58.352320 21408 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 01:10:58.352418 21408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:10:58.352507 21408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:58.352663 21408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:58.352718 21408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:10:58.352787 21408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:10:58.362573 21408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:38197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:38197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:58.363838 21408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:58.365185 21408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:58.377465 21414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:58.377511 21413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:58.377556 21416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:58.378199 21408 server_base.cc:1034] running on GCE node
I20260929 01:10:58.378969 21408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:58.380527 21408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:58.381778 21408 hybrid_clock.cc:648] HybridClock initialized: now 1790644258381723 us; error 34 us; skew 500 ppm
I20260929 01:10:58.382102 21408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:58 dist-test-slave-41cl krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644258, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:58.385152 21408 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:10:58.386565 21408 webserver.cc:460] Webserver started at http://127.5.198.254:43013/ using document root <none> and password file <none>
I20260929 01:10:58.387238 21408 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:58.387367 21408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:58.387681 21408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:10:58.390610 21408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "3f4f2894592c4439b04146fe0419db90"
format_stamp: "Formatted at 2026-09-29 01:10:58 on dist-test-slave-41cl"
server_key: "f4eb7aadaddcdf67f10003e09c2add67"
server_key_iv: "922cd4e9b098a4174f45183176ab8515"
server_key_version: "encryptionkey@0"
I20260929 01:10:58.391394 21408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "3f4f2894592c4439b04146fe0419db90"
format_stamp: "Formatted at 2026-09-29 01:10:58 on dist-test-slave-41cl"
server_key: "f4eb7aadaddcdf67f10003e09c2add67"
server_key_iv: "922cd4e9b098a4174f45183176ab8515"
server_key_version: "encryptionkey@0"
I20260929 01:10:58.396308 21408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260929 01:10:58.399571 21423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:58.400921 21408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:10:58.401121 21408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "3f4f2894592c4439b04146fe0419db90"
format_stamp: "Formatted at 2026-09-29 01:10:58 on dist-test-slave-41cl"
server_key: "f4eb7aadaddcdf67f10003e09c2add67"
server_key_iv: "922cd4e9b098a4174f45183176ab8515"
server_key_version: "encryptionkey@0"
I20260929 01:10:58.401338 21408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:58.439769 21408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:58.443892 21408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:58.444185 21408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:58.463114 21408 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:38197
I20260929 01:10:58.463138 21474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:38197 every 8 connection(s)
I20260929 01:10:58.464406 21408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:10:58.467523 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 21408
I20260929 01:10:58.467736 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:10:58.468626 5915 external_mini_cluster.cc:1428] Setting key dec1508787f6f54ddb2a29cab600f74d
I20260929 01:10:58.468768 21475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:10:58.481142 21475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Bootstrap starting.
Sep 29 01:10:58 dist-test-slave-41cl krb5kdc[21395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:58.484524 21475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Neither blocks nor log segments found. Creating new log.
I20260929 01:10:58.485728 21475 log.cc:824] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Log is configured to *not* fsync() on all Append() calls
I20260929 01:10:58.488431 21475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: No bootstrap required, opened a new log
I20260929 01:10:58.490427 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:58.470733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55264 (local address 127.5.198.254:38197)
0929 01:10:58.471470 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:58.471481 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:58.471519 (+ 38us) server_negotiation.cc:407] Connection header received
0929 01:10:58.473544 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:58.473564 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:58.473920 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:58.474281 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:58.475718 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:58.477009 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:58.478351 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:58.479023 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:58.479247 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:58.483397 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:58.483405 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:58.483419 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:58.483456 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:58.486591 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:58.487286 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:58.487293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:58.487299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:58.487453 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:58.487843 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:58.487848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:58.487851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:58.488332 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:58.488657 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:58.489123 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 01:10:58.489511 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":328,"threads_started":1}
I20260929 01:10:58.496573 21475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } }
I20260929 01:10:58.496968 21475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:10:58.497076 21475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4f2894592c4439b04146fe0419db90, State: Initialized, Role: FOLLOWER
I20260929 01:10:58.497637 21475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } }
I20260929 01:10:58.497891 21475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:10:58.498032 21475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:10:58.498196 21475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:10:58.500741 21475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } }
I20260929 01:10:58.501204 21475 leader_election.cc:302] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f4f2894592c4439b04146fe0419db90; no voters:
I20260929 01:10:58.501775 21475 leader_election.cc:288] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:10:58.501904 21480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:10:58.502600 21480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 1 LEADER]: Becoming Leader. State: Replica: 3f4f2894592c4439b04146fe0419db90, State: Running, Role: LEADER
I20260929 01:10:58.503168 21480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } }
I20260929 01:10:58.503779 21475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:10:58.505710 21482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f4f2894592c4439b04146fe0419db90. Latest consensus state: current_term: 1 leader_uuid: "3f4f2894592c4439b04146fe0419db90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } } }
I20260929 01:10:58.506102 21482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:58.506695 21481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f4f2894592c4439b04146fe0419db90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } } }
I20260929 01:10:58.507009 21481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [sys.catalog]: This master's current role is: LEADER
I20260929 01:10:58.507098 21486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:10:58.515733 21486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:10:58.530499 21486 catalog_manager.cc:1347] Generated new cluster ID: 8d463c170fea4ff0a16634f7f6ee261c
I20260929 01:10:58.530637 21486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:10:58.556330 21486 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:10:58.557704 21486 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:10:58.564687 21486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Generated new TSK 0
I20260929 01:10:58.565474 21486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:10:58.637411 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:38197
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:10:58.970484 21502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:58.970762 21502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:58.970929 21502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:58.978147 21502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:58.978250 21502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:58.978459 21502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:10:58.988997 21502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:58.990254 21502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:58.991524 21502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:59.004424 21508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:59.004539 21510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:59.004554 21507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:59.005990 21502 server_base.cc:1034] running on GCE node
I20260929 01:10:59.006574 21502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:59.007524 21502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:59.008752 21502 hybrid_clock.cc:648] HybridClock initialized: now 1790644259008691 us; error 50 us; skew 500 ppm
I20260929 01:10:59.009061 21502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:59 dist-test-slave-41cl krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644259, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:59.013110 21502 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:10:59.014513 21502 webserver.cc:460] Webserver started at http://127.5.198.193:36061/ using document root <none> and password file <none>
I20260929 01:10:59.015211 21502 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:59.015363 21502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:59.015686 21502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:10:59.018785 21502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "abff705fff9346c9a09fb1dc9c46af69"
format_stamp: "Formatted at 2026-09-29 01:10:59 on dist-test-slave-41cl"
server_key: "ff97f94e0a3a05b1fd0d49afed271a54"
server_key_iv: "5dc21860773cf0ab4766e4e2a3dfa77b"
server_key_version: "encryptionkey@0"
I20260929 01:10:59.019569 21502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "abff705fff9346c9a09fb1dc9c46af69"
format_stamp: "Formatted at 2026-09-29 01:10:59 on dist-test-slave-41cl"
server_key: "ff97f94e0a3a05b1fd0d49afed271a54"
server_key_iv: "5dc21860773cf0ab4766e4e2a3dfa77b"
server_key_version: "encryptionkey@0"
I20260929 01:10:59.024583 21502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 01:10:59.027910 21517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:59.029285 21502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:10:59.029465 21502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "abff705fff9346c9a09fb1dc9c46af69"
format_stamp: "Formatted at 2026-09-29 01:10:59 on dist-test-slave-41cl"
server_key: "ff97f94e0a3a05b1fd0d49afed271a54"
server_key_iv: "5dc21860773cf0ab4766e4e2a3dfa77b"
server_key_version: "encryptionkey@0"
I20260929 01:10:59.029677 21502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:59.051040 21502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:59.055460 21502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:59.055763 21502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:59.056815 21502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:59.058174 21502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:10:59.058280 21502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:59.058432 21502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:10:59.058506 21502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:59.091753 21502 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:37583
I20260929 01:10:59.091813 21629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:37583 every 8 connection(s)
I20260929 01:10:59.093135 21502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:10:59.101037 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 21502
I20260929 01:10:59.101244 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:10:59.101995 5915 external_mini_cluster.cc:1428] Setting key d5bdd36420102f9bd7276385c70d307e
Sep 29 01:10:59 dist-test-slave-41cl krb5kdc[21395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644259, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:59.114125 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:59.095748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:53703 (local address 127.5.198.254:38197)
0929 01:10:59.096279 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:59.096283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:59.097085 (+ 802us) server_negotiation.cc:407] Connection header received
0929 01:10:59.098845 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:59.098853 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:59.098946 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:59.099073 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:59.100487 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:59.101640 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:59.103398 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:59.103973 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:59.104090 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:59.108186 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:59.108191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:59.108194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:59.108226 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:59.110975 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:59.111720 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:59.111727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:59.111730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:59.111858 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:59.112291 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:59.112297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:59.112300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:59.112643 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:59.112797 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:59.113524 (+ 727us) server_negotiation.cc:299] Negotiation successful
0929 01:10:59.113747 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":306,"threads_started":1}
I20260929 01:10:59.115065 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:59.096097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:53703)
0929 01:10:59.096884 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:59.096928 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:59.098589 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:59.099287 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:59.099294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:59.099781 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:59.100304 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:59.100321 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:59.101797 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:59.101801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:59.103240 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:59.103252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:59.104225 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:59.104233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:59.104549 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:59.105309 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:59.105344 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:59.107960 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:59.111129 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:59.111138 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:59.111154 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:59.111590 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:59.111972 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:59.111985 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:59.111989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:59.112176 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:59.112778 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:59.112786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:59.113132 (+ 346us) client_negotiation.cc:769] Sending connection context
0929 01:10:59.113411 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 01:10:59.113933 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":378,"threads_started":1}
I20260929 01:10:59.116762 21630 heartbeater.cc:347] Connected to a master server at 127.5.198.254:38197
I20260929 01:10:59.117079 21630 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:59.117795 21630 heartbeater.cc:511] Master 127.5.198.254:38197 requested a full tablet report, sending...
I20260929 01:10:59.120164 21440 ts_manager.cc:194] Registered new tserver with Master: abff705fff9346c9a09fb1dc9c46af69 (127.5.198.193:37583)
I20260929 01:10:59.122395 21440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:53703
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:10:59.156584 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:38197
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:10:59.479876 21636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:59.480242 21636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:59.480381 21636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:59.490461 21636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:59.490576 21636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:59.490692 21636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:10:59.500360 21636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:10:59.501631 21636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:10:59.502887 21636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:10:59.516605 21644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:59.516680 21642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:10:59.516703 21641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:10:59.517221 21636 server_base.cc:1034] running on GCE node
I20260929 01:10:59.518061 21636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:10:59.519110 21636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:10:59.520328 21636 hybrid_clock.cc:648] HybridClock initialized: now 1790644259520279 us; error 41 us; skew 500 ppm
I20260929 01:10:59.520752 21636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:10:59 dist-test-slave-41cl krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644259, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:10:59.524082 21636 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:10:59.525687 21636 webserver.cc:460] Webserver started at http://127.5.198.194:41895/ using document root <none> and password file <none>
I20260929 01:10:59.526371 21636 fs_manager.cc:359] Metadata directory not provided
I20260929 01:10:59.526520 21636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:10:59.526840 21636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:10:59.529840 21636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "ef8ca4430404475a90195e362b3bae28"
format_stamp: "Formatted at 2026-09-29 01:10:59 on dist-test-slave-41cl"
server_key: "98f98ec1fb99500ef655dccfa8b4c636"
server_key_iv: "10d2d537ea6fae3f7e2623cfb1162b06"
server_key_version: "encryptionkey@0"
I20260929 01:10:59.530617 21636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "ef8ca4430404475a90195e362b3bae28"
format_stamp: "Formatted at 2026-09-29 01:10:59 on dist-test-slave-41cl"
server_key: "98f98ec1fb99500ef655dccfa8b4c636"
server_key_iv: "10d2d537ea6fae3f7e2623cfb1162b06"
server_key_version: "encryptionkey@0"
I20260929 01:10:59.535985 21636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 01:10:59.539623 21651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:59.541024 21636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:10:59.541216 21636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "ef8ca4430404475a90195e362b3bae28"
format_stamp: "Formatted at 2026-09-29 01:10:59 on dist-test-slave-41cl"
server_key: "98f98ec1fb99500ef655dccfa8b4c636"
server_key_iv: "10d2d537ea6fae3f7e2623cfb1162b06"
server_key_version: "encryptionkey@0"
I20260929 01:10:59.541435 21636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:10:59.556814 21636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:10:59.560585 21636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:10:59.560873 21636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:10:59.561792 21636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:10:59.563061 21636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:10:59.563164 21636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:59.563323 21636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:10:59.563396 21636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:10:59.596549 21636 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:40439
I20260929 01:10:59.596607 21763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:40439 every 8 connection(s)
I20260929 01:10:59.597911 21636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:10:59.598515 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 21636
I20260929 01:10:59.598726 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:10:59.599383 5915 external_mini_cluster.cc:1428] Setting key b2d3a4ebd1b37a24dc7ff6e5829eec1c
Sep 29 01:10:59 dist-test-slave-41cl krb5kdc[21395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644259, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:10:59.624966 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:59.600659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:48567 (local address 127.5.198.254:38197)
0929 01:10:59.600865 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 01:10:59.600872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:10:59.602676 (+ 1804us) server_negotiation.cc:407] Connection header received
0929 01:10:59.605521 (+ 2845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:10:59.605530 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:10:59.605643 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:10:59.605763 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:10:59.607737 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:59.608975 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:59.610692 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:10:59.611192 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:10:59.611297 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:59.616555 (+ 5258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:10:59.616579 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:10:59.616582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:10:59.616621 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:10:59.620659 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:59.621492 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:59.621501 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:59.621507 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:59.621682 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:10:59.622349 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:10:59.622356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:10:59.622360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:10:59.622781 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:10:59.623278 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0929 01:10:59.624087 (+ 809us) server_negotiation.cc:299] Negotiation successful
0929 01:10:59.624430 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:10:59.625554 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:10:59.601415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:48567)
0929 01:10:59.602501 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0929 01:10:59.602552 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:10:59.605177 (+ 2625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:10:59.606090 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:10:59.606099 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:10:59.606660 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:10:59.607521 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:59.607543 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:59.609208 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:59.609212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:59.610524 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:10:59.610536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:10:59.611603 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:10:59.611616 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:10:59.611968 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:10:59.612883 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:10:59.612917 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:10:59.616208 (+ 3291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:10:59.620841 (+ 4633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:59.620857 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:59.620872 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:59.621348 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:59.621851 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:10:59.621857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:10:59.621860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:10:59.622175 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:10:59.623206 (+ 1031us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:10:59.623213 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:10:59.623602 (+ 389us) client_negotiation.cc:769] Sending connection context
0929 01:10:59.623884 (+ 282us) client_negotiation.cc:240] Negotiation successful
0929 01:10:59.624299 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":560,"threads_started":1}
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
I20260929 01:10:59.627302 21764 heartbeater.cc:347] Connected to a master server at 127.5.198.254:38197
I20260929 01:10:59.627661 21764 heartbeater.cc:464] Registering TS with master...
I20260929 01:10:59.628652 21764 heartbeater.cc:511] Master 127.5.198.254:38197 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
I20260929 01:10:59.630367 21440 ts_manager.cc:194] Registered new tserver with Master: ef8ca4430404475a90195e362b3bae28 (127.5.198.194:40439)
I20260929 01:10:59.631662 21440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:48567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:10:59.660200 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:38197
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:10:59.983806 21769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:10:59.984189 21769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:10:59.984359 21769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:10:59.991680 21769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:10:59.991842 21769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:10:59.992023 21769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:11:00.002059 21769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:11:00.003401 21769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:11:00.004851 21769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:11:00.019932 21774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:00.020144 21777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:00.020219 21775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:11:00.020817 21769 server_base.cc:1034] running on GCE node
I20260929 01:11:00.021520 21769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:11:00.022535 21769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:11:00.023751 21769 hybrid_clock.cc:648] HybridClock initialized: now 1790644260023703 us; error 38 us; skew 500 ppm
I20260929 01:11:00.024096 21769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:11:00 dist-test-slave-41cl krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644260, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:11:00.027366 21769 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:11:00.028853 21769 webserver.cc:460] Webserver started at http://127.5.198.195:36103/ using document root <none> and password file <none>
I20260929 01:11:00.029518 21769 fs_manager.cc:359] Metadata directory not provided
I20260929 01:11:00.029618 21769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:11:00.029881 21769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:11:00.032959 21769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "80f5a55a3a2442dab20236503cef7729"
format_stamp: "Formatted at 2026-09-29 01:11:00 on dist-test-slave-41cl"
server_key: "064fd0f3193fe7d19769e3294bdf7ebd"
server_key_iv: "d5475b670a9da224a8df2fa0d0c38b36"
server_key_version: "encryptionkey@0"
I20260929 01:11:00.033751 21769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "80f5a55a3a2442dab20236503cef7729"
format_stamp: "Formatted at 2026-09-29 01:11:00 on dist-test-slave-41cl"
server_key: "064fd0f3193fe7d19769e3294bdf7ebd"
server_key_iv: "d5475b670a9da224a8df2fa0d0c38b36"
server_key_version: "encryptionkey@0"
I20260929 01:11:00.038993 21769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 01:11:00.042634 21784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:00.044029 21769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:11:00.044207 21769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "80f5a55a3a2442dab20236503cef7729"
format_stamp: "Formatted at 2026-09-29 01:11:00 on dist-test-slave-41cl"
server_key: "064fd0f3193fe7d19769e3294bdf7ebd"
server_key_iv: "d5475b670a9da224a8df2fa0d0c38b36"
server_key_version: "encryptionkey@0"
I20260929 01:11:00.044422 21769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:11:00.064539 21769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:11:00.068900 21769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:11:00.069205 21769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:11:00.070160 21769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:11:00.071466 21769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:11:00.071576 21769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:00.071730 21769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:11:00.071813 21769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:00.105602 21769 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:45563
I20260929 01:11:00.105618 21896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:45563 every 8 connection(s)
I20260929 01:11:00.106851 21769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:11:00.113145 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 21769
I20260929 01:11:00.113351 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:11:00.113899 5915 external_mini_cluster.cc:1428] Setting key 2c65fad93315cdfbbd43c90361f55497
Sep 29 01:11:00 dist-test-slave-41cl krb5kdc[21395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644260, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:11:00.125505 21630 heartbeater.cc:503] Master 127.5.198.254:38197 was elected leader, sending a full tablet report...
I20260929 01:11:00.128832 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.109418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:47889 (local address 127.5.198.254:38197)
0929 01:11:00.109638 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:00.109644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:00.110761 (+ 1117us) server_negotiation.cc:407] Connection header received
0929 01:11:00.112539 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:00.112543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:00.112636 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:00.112740 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:11:00.114804 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.115935 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.117811 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.118521 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.118627 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.122399 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:11:00.122404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:11:00.122408 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:11:00.122442 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:11:00.125113 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:00.126067 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:00.126073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:00.126076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:00.126224 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:00.126797 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:00.126801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:00.126803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:00.127154 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:11:00.127323 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:00.128199 (+ 876us) server_negotiation.cc:299] Negotiation successful
0929 01:11:00.128431 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 01:11:00.129999 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.109848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:47889)
0929 01:11:00.110597 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:00.110642 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 01:11:00.112240 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:11:00.113014 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:11:00.113024 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:11:00.113798 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:11:00.114589 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:00.114610 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.116111 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.116115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:00.117677 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:00.117689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.118731 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.118737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:00.118941 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.119719 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:11:00.119746 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:11:00.122190 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:11:00.125320 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:11:00.125330 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:11:00.125347 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:11:00.125948 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:11:00.126398 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:11:00.126403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:11:00.126406 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:11:00.126680 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:11:00.127299 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:11:00.127307 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:11:00.127743 (+ 436us) client_negotiation.cc:769] Sending connection context
0929 01:11:00.128070 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 01:11:00.128699 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":324,"threads_started":1}
I20260929 01:11:00.131330 21897 heartbeater.cc:347] Connected to a master server at 127.5.198.254:38197
I20260929 01:11:00.131697 21897 heartbeater.cc:464] Registering TS with master...
I20260929 01:11:00.132473 21897 heartbeater.cc:511] Master 127.5.198.254:38197 requested a full tablet report, sending...
I20260929 01:11:00.134160 21440 ts_manager.cc:194] Registered new tserver with Master: 80f5a55a3a2442dab20236503cef7729 (127.5.198.195:45563)
I20260929 01:11:00.135010 21440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:47889
I20260929 01:11:00.142205 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:11:00.159497 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.146853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55278 (local address 127.5.198.254:38197)
0929 01:11:00.147011 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:00.147015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:00.147416 (+ 401us) server_negotiation.cc:407] Connection header received
0929 01:11:00.147600 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:00.147603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:00.147663 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:00.147744 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:11:00.149317 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.150120 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.151549 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.151981 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.152085 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.154561 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:11:00.154567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:11:00.154570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:11:00.154602 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:11:00.157256 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:00.157877 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:00.157880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:00.157883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:00.157997 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:00.158342 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:00.158345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:00.158347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:00.158667 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:11:00.158841 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:00.159112 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 01:11:00.159268 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:11:00.163852 21440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:11:00.170912 21440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:11:00.193368 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.183294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37583 (local address 127.0.0.1:37018)
0929 01:11:00.184242 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:00.184280 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:11:00.184490 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:11:00.185834 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:11:00.185841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:11:00.185871 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:11:00.186282 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:00.186297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.188559 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.188564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:00.190810 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:00.190824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.192141 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.192146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:00.192454 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.192564 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 01:11:00.192708 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 01:11:00.192906 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":124,"thread_start_us":480,"threads_started":1}
I20260929 01:11:00.194382 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.183622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37018 (local address 127.5.198.193:37583)
0929 01:11:00.185165 (+ 1543us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:00.185177 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:00.185207 (+ 30us) server_negotiation.cc:407] Connection header received
0929 01:11:00.185317 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:00.185325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:00.185594 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:00.186505 (+ 911us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:11:00.186725 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.188394 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.190978 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.191935 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.192102 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.192347 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:00.192758 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 01:11:00.192918 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1372,"thread_start_us":1236,"threads_started":1}
I20260929 01:11:00.195504 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.183489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:45563 (local address 127.0.0.1:56368)
0929 01:11:00.185417 (+ 1928us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:00.185434 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:11:00.185534 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:11:00.187083 (+ 1549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:11:00.187088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:11:00.187112 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:11:00.187588 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:00.187601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.189832 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.189837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:00.192609 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:00.192624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.194656 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.194662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:00.194843 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.194883 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 01:11:00.194949 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 01:11:00.195084 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1817,"mutex_wait_us":159,"thread_start_us":3374,"threads_started":1}
I20260929 01:11:00.195662 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.184060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:40439 (local address 127.0.0.1:58814)
0929 01:11:00.186773 (+ 2713us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:00.186791 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:11:00.186902 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:11:00.187794 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:11:00.187798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:11:00.187823 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:11:00.188155 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:00.188165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.189986 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.189991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:00.192611 (+ 2620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:00.192624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.194874 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.194879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:00.195046 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.195074 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 01:11:00.195155 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 01:11:00.195290 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2578,"mutex_wait_us":97,"thread_start_us":1112,"threads_started":1}
I20260929 01:11:00.195729 21914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.184826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.5.198.194:40439)
0929 01:11:00.186033 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:00.186041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:00.187037 (+ 996us) server_negotiation.cc:407] Connection header received
0929 01:11:00.187135 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:00.187142 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:00.187440 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:00.187716 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:11:00.188321 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.189817 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.192801 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.194388 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.194533 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.194740 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:00.195232 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 01:11:00.195435 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":912,"threads_started":1}
I20260929 01:11:00.197273 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.184524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56368 (local address 127.5.198.195:45563)
0929 01:11:00.186512 (+ 1988us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:00.186519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:00.186549 (+ 30us) server_negotiation.cc:407] Connection header received
0929 01:11:00.186639 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:00.186645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:00.186855 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:00.187716 (+ 861us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:11:00.188009 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.189663 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.192802 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.194376 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.196627 (+ 2251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.196807 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:00.196861 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 01:11:00.196989 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1827,"thread_start_us":1663,"threads_started":1}
I20260929 01:11:00.198446 21565 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5c45460ad947b4bdafbc4368d1a444 (DEFAULT_TABLE table=test-table [id=59ea528ad04b4099842457b518f86811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:11:00.199198 21699 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5c45460ad947b4bdafbc4368d1a444 (DEFAULT_TABLE table=test-table [id=59ea528ad04b4099842457b518f86811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:11:00.200057 21565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5c45460ad947b4bdafbc4368d1a444. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:11:00.200565 21699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5c45460ad947b4bdafbc4368d1a444. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:11:00.202889 21832 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5c45460ad947b4bdafbc4368d1a444 (DEFAULT_TABLE table=test-table [id=59ea528ad04b4099842457b518f86811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:11:00.204267 21832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5c45460ad947b4bdafbc4368d1a444. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:11:00.216434 21916 tablet_bootstrap.cc:492] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69: Bootstrap starting.
I20260929 01:11:00.217739 21917 tablet_bootstrap.cc:492] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729: Bootstrap starting.
I20260929 01:11:00.218086 21915 tablet_bootstrap.cc:492] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28: Bootstrap starting.
I20260929 01:11:00.220438 21916 tablet_bootstrap.cc:654] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69: Neither blocks nor log segments found. Creating new log.
I20260929 01:11:00.220739 21917 tablet_bootstrap.cc:654] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729: Neither blocks nor log segments found. Creating new log.
I20260929 01:11:00.221482 21915 tablet_bootstrap.cc:654] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28: Neither blocks nor log segments found. Creating new log.
I20260929 01:11:00.221750 21917 log.cc:824] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729: Log is configured to *not* fsync() on all Append() calls
I20260929 01:11:00.221769 21916 log.cc:824] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69: Log is configured to *not* fsync() on all Append() calls
I20260929 01:11:00.222779 21915 log.cc:824] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28: Log is configured to *not* fsync() on all Append() calls
I20260929 01:11:00.224944 21917 tablet_bootstrap.cc:492] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729: No bootstrap required, opened a new log
I20260929 01:11:00.225298 21917 ts_tablet_manager.cc:1397] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260929 01:11:00.225910 21916 tablet_bootstrap.cc:492] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69: No bootstrap required, opened a new log
I20260929 01:11:00.225925 21915 tablet_bootstrap.cc:492] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28: No bootstrap required, opened a new log
I20260929 01:11:00.226186 21916 ts_tablet_manager.cc:1397] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260929 01:11:00.226186 21915 ts_tablet_manager.cc:1397] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20260929 01:11:00.232918 21917 raft_consensus.cc:348] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:00.233364 21917 raft_consensus.cc:374] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:11:00.233492 21917 raft_consensus.cc:729] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f5a55a3a2442dab20236503cef7729, State: Initialized, Role: FOLLOWER
I20260929 01:11:00.234259 21917 consensus_queue.cc:260] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:00.236078 21917 ts_tablet_manager.cc:1428] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260929 01:11:00.236970 21897 heartbeater.cc:503] Master 127.5.198.254:38197 was elected leader, sending a full tablet report...
I20260929 01:11:00.237222 21916 raft_consensus.cc:348] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:00.237221 21915 raft_consensus.cc:348] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:00.237673 21915 raft_consensus.cc:374] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:11:00.237674 21916 raft_consensus.cc:374] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:11:00.237782 21915 raft_consensus.cc:729] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8ca4430404475a90195e362b3bae28, State: Initialized, Role: FOLLOWER
I20260929 01:11:00.237782 21916 raft_consensus.cc:729] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abff705fff9346c9a09fb1dc9c46af69, State: Initialized, Role: FOLLOWER
I20260929 01:11:00.238409 21916 consensus_queue.cc:260] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:00.238420 21915 consensus_queue.cc:260] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:00.239936 21764 heartbeater.cc:503] Master 127.5.198.254:38197 was elected leader, sending a full tablet report...
I20260929 01:11:00.240236 21916 ts_tablet_manager.cc:1428] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260929 01:11:00.240563 21915 ts_tablet_manager.cc:1428] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
W20260929 01:11:00.345520 21631 tablet.cc:2369] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:11:00.350546 21765 tablet.cc:2369] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:11:00.358505 21898 tablet.cc:2369] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:11:00.380965 21921 raft_consensus.cc:482] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:11:00.381248 21921 raft_consensus.cc:504] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:00.382874 21921 leader_election.cc:288] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abff705fff9346c9a09fb1dc9c46af69 (127.5.198.193:37583), ef8ca4430404475a90195e362b3bae28 (127.5.198.194:40439)
I20260929 01:11:00.389531 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.383147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:40439 (local address 127.5.198.195:48301)
0929 01:11:00.383359 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:00.383381 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:11:00.383510 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:11:00.384001 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:11:00.384007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:11:00.384041 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:11:00.384290 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:00.384299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.385494 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.385498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:00.387395 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:00.387410 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.388833 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.388837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:00.388985 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.389009 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 01:11:00.389136 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 01:11:00.389230 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"spinlock_wait_cycles":30208}
I20260929 01:11:00.389590 21914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.383407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:48301 (local address 127.5.198.194:40439)
0929 01:11:00.383660 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:00.383669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:00.383698 (+ 29us) server_negotiation.cc:407] Connection header received
0929 01:11:00.383773 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:00.383778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:00.383859 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:00.383942 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:11:00.384441 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.385367 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.387571 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.388671 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.388826 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.388907 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:00.389183 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 01:11:00.389315 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 01:11:00.390419 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.383332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:54241 (local address 127.5.198.193:37583)
0929 01:11:00.383525 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:00.383531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:00.384115 (+ 584us) server_negotiation.cc:407] Connection header received
0929 01:11:00.384266 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:00.384269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:00.384325 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:00.384433 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:11:00.385074 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.385887 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.388018 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.389362 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.389526 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.389599 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:00.390057 (+ 458us) server_negotiation.cc:299] Negotiation successful
0929 01:11:00.390184 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 01:11:00.390506 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.383147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37583 (local address 127.5.198.195:54241)
0929 01:11:00.383993 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:00.384009 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:11:00.384149 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:11:00.384502 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:11:00.384506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:11:00.384534 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:11:00.384867 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:00.384878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.386022 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.386027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:00.387870 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:00.387880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.389630 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.389639 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:00.389827 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.389867 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 01:11:00.389981 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 01:11:00.390106 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":238,"spinlock_wait_cycles":768,"thread_start_us":286,"threads_started":1}
I20260929 01:11:00.390745 21719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5c45460ad947b4bdafbc4368d1a444" candidate_uuid: "80f5a55a3a2442dab20236503cef7729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef8ca4430404475a90195e362b3bae28" is_pre_election: true
I20260929 01:11:00.391275 21719 raft_consensus.cc:2454] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80f5a55a3a2442dab20236503cef7729 in term 0.
I20260929 01:11:00.391377 21585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5c45460ad947b4bdafbc4368d1a444" candidate_uuid: "80f5a55a3a2442dab20236503cef7729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abff705fff9346c9a09fb1dc9c46af69" is_pre_election: true
I20260929 01:11:00.391912 21785 leader_election.cc:302] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80f5a55a3a2442dab20236503cef7729, ef8ca4430404475a90195e362b3bae28; no voters:
I20260929 01:11:00.391906 21585 raft_consensus.cc:2454] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80f5a55a3a2442dab20236503cef7729 in term 0.
I20260929 01:11:00.392297 21921 raft_consensus.cc:2781] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:11:00.392499 21921 raft_consensus.cc:482] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:11:00.392624 21921 raft_consensus.cc:3037] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:11:00.394802 21921 raft_consensus.cc:504] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:00.395442 21921 leader_election.cc:288] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [CANDIDATE]: Term 1 election: Requested vote from peers abff705fff9346c9a09fb1dc9c46af69 (127.5.198.193:37583), ef8ca4430404475a90195e362b3bae28 (127.5.198.194:40439)
I20260929 01:11:00.395798 21585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5c45460ad947b4bdafbc4368d1a444" candidate_uuid: "80f5a55a3a2442dab20236503cef7729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abff705fff9346c9a09fb1dc9c46af69"
I20260929 01:11:00.396021 21585 raft_consensus.cc:3037] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:11:00.395952 21719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5c45460ad947b4bdafbc4368d1a444" candidate_uuid: "80f5a55a3a2442dab20236503cef7729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef8ca4430404475a90195e362b3bae28"
I20260929 01:11:00.396206 21719 raft_consensus.cc:3037] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:11:00.398352 21719 raft_consensus.cc:2454] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80f5a55a3a2442dab20236503cef7729 in term 1.
I20260929 01:11:00.398388 21585 raft_consensus.cc:2454] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80f5a55a3a2442dab20236503cef7729 in term 1.
I20260929 01:11:00.398753 21785 leader_election.cc:302] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80f5a55a3a2442dab20236503cef7729, ef8ca4430404475a90195e362b3bae28; no voters:
I20260929 01:11:00.399037 21921 raft_consensus.cc:2781] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:11:00.399464 21921 raft_consensus.cc:686] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [term 1 LEADER]: Becoming Leader. State: Replica: 80f5a55a3a2442dab20236503cef7729, State: Running, Role: LEADER
I20260929 01:11:00.399914 21921 consensus_queue.cc:237] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:00.404835 21440 catalog_manager.cc:5495] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80f5a55a3a2442dab20236503cef7729 (127.5.198.195). New cstate: current_term: 1 leader_uuid: "80f5a55a3a2442dab20236503cef7729" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } health_report { overall_health: UNKNOWN } } }
I20260929 01:11:00.436618 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.430670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.5.198.195:45563)
0929 01:11:00.430891 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:00.430897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:00.430917 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:11:00.431121 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:00.431126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:00.431219 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:00.431371 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:11:00.431750 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.432613 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.434375 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.434781 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.434919 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.435448 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:11:00.435960 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:11:00.436146 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:00.436197 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 01:11:00.436280 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 01:11:00.445333 21585 raft_consensus.cc:1261] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Refusing update from remote peer 80f5a55a3a2442dab20236503cef7729: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:11:00.445334 21719 raft_consensus.cc:1261] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28 [term 1 FOLLOWER]: Refusing update from remote peer 80f5a55a3a2442dab20236503cef7729: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:11:00.446466 21925 consensus_queue.cc:1035] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:11:00.446866 21921 consensus_queue.cc:1035] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:11:00.460517 21928 mvcc.cc:205] Tried to move back new op lower bound from 7334478890772021248 to 7334478890602299392. Current Snapshot: MvccSnapshot[applied={T|T < 7334478890772021248}]
I20260929 01:11:00.461163 21929 mvcc.cc:205] Tried to move back new op lower bound from 7334478890772021248 to 7334478890602299392. Current Snapshot: MvccSnapshot[applied={T|T < 7334478890772021248}]
I20260929 01:11:00.468351 21930 mvcc.cc:205] Tried to move back new op lower bound from 7334478890772021248 to 7334478890602299392. Current Snapshot: MvccSnapshot[applied={T|T < 7334478890772021248 or (T in {7334478890772021248})}]
I20260929 01:11:00.480705 21440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:11:00.480978 21440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:11:00.484730 21440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Sending DeleteTablet for 3 replicas of tablet 0d5c45460ad947b4bdafbc4368d1a444
I20260929 01:11:00.485937 21565 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5c45460ad947b4bdafbc4368d1a444 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:11:00 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:37018
I20260929 01:11:00.486317 21699 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5c45460ad947b4bdafbc4368d1a444 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:11:00 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:58814
I20260929 01:11:00.486965 21832 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5c45460ad947b4bdafbc4368d1a444 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:11:00 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:56368
I20260929 01:11:00.488044 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 21502
I20260929 01:11:00.496767 21938 tablet_replica.cc:330] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69: stopping tablet replica
I20260929 01:11:00.497288 21938 raft_consensus.cc:2229] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:11:00.497910 21938 raft_consensus.cc:2258] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:11:00.500205 21938 ts_tablet_manager.cc:1905] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:11:00.501852 21939 tablet_replica.cc:330] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28: stopping tablet replica
I20260929 01:11:00.502511 21939 raft_consensus.cc:2229] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:11:00.503160 21939 raft_consensus.cc:2258] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:11:00.506011 21939 ts_tablet_manager.cc:1905] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:11:00.507577 21938 ts_tablet_manager.cc:1918] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:11:00.507773 21938 log.cc:1196] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/0d5c45460ad947b4bdafbc4368d1a444
I20260929 01:11:00.508361 21938 ts_tablet_manager.cc:1939] T 0d5c45460ad947b4bdafbc4368d1a444 P abff705fff9346c9a09fb1dc9c46af69: Deleting consensus metadata
I20260929 01:11:00.510278 21425 catalog_manager.cc:4841] TS abff705fff9346c9a09fb1dc9c46af69 (127.5.198.193:37583): tablet 0d5c45460ad947b4bdafbc4368d1a444 (table test-table [id=59ea528ad04b4099842457b518f86811]) successfully deleted
I20260929 01:11:00.511857 21940 tablet_replica.cc:330] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729: stopping tablet replica
I20260929 01:11:00.512338 21940 raft_consensus.cc:2229] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:11:00.513190 21940 raft_consensus.cc:2258] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:11:00.514853 21939 ts_tablet_manager.cc:1918] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:11:00.515066 21939 log.cc:1196] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/0d5c45460ad947b4bdafbc4368d1a444
I20260929 01:11:00.515314 21940 ts_tablet_manager.cc:1905] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:11:00.515655 21939 ts_tablet_manager.cc:1939] T 0d5c45460ad947b4bdafbc4368d1a444 P ef8ca4430404475a90195e362b3bae28: Deleting consensus metadata
I20260929 01:11:00.517602 21424 catalog_manager.cc:4841] TS ef8ca4430404475a90195e362b3bae28 (127.5.198.194:40439): tablet 0d5c45460ad947b4bdafbc4368d1a444 (table test-table [id=59ea528ad04b4099842457b518f86811]) successfully deleted
Sep 29 01:11:00 dist-test-slave-41cl krb5kdc[21395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:11:00.522955 21940 ts_tablet_manager.cc:1918] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:11:00.523124 21940 log.cc:1196] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/0d5c45460ad947b4bdafbc4368d1a444
I20260929 01:11:00.523689 21940 ts_tablet_manager.cc:1939] T 0d5c45460ad947b4bdafbc4368d1a444 P 80f5a55a3a2442dab20236503cef7729: Deleting consensus metadata
I20260929 01:11:00.525794 21427 catalog_manager.cc:4841] TS 80f5a55a3a2442dab20236503cef7729 (127.5.198.195:45563): tablet 0d5c45460ad947b4bdafbc4368d1a444 (table test-table [id=59ea528ad04b4099842457b518f86811]) successfully deleted
I20260929 01:11:00.530210 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.489652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37026 (local address 127.5.198.193:37583)
0929 01:11:00.489908 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:00.489919 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:00.511165 (+ 21246us) server_negotiation.cc:407] Connection header received
0929 01:11:00.511393 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:00.511398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:00.511485 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:00.511600 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:11:00.514291 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.515525 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.517903 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.518507 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.518661 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.523511 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:11:00.523525 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:11:00.523539 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:11:00.523573 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:11:00.527226 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:00.528146 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:00.528153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:00.528156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:00.528289 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:00.528730 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:00.528734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:00.528736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:00.529112 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:11:00.529335 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:00.529711 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 01:11:00.529872 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 01:11:00.530937 21625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:11:00.633253 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 21502
I20260929 01:11:00.652134 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 21636
Sep 29 01:11:00 dist-test-slave-41cl krb5kdc[21395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:11:00.670800 21914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.653022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58820 (local address 127.5.198.194:40439)
0929 01:11:00.653195 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:00.653199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:00.653218 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:11:00.653570 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:00.653576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:00.653714 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:00.653851 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:11:00.655826 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.656738 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.658705 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.659114 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.659233 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.664932 (+ 5699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:11:00.664948 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:11:00.664963 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:11:00.665003 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:11:00.667902 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:00.668683 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:00.668690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:00.668693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:00.668820 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:00.669223 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:00.669229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:00.669232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:00.669610 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:11:00.669912 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:00.670227 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 01:11:00.670464 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:11:00.671490 21759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:11:00.761953 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 21636
I20260929 01:11:00.779685 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 21769
Sep 29 01:11:00 dist-test-slave-41cl krb5kdc[21395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:11:00.798015 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:00.780487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.5.198.195:45563)
0929 01:11:00.780678 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:00.780686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:00.780715 (+ 29us) server_negotiation.cc:407] Connection header received
0929 01:11:00.780825 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:00.780831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:00.780927 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:00.781047 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:11:00.782371 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.783610 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.785091 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:00.785781 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:00.785917 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:00.790553 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:11:00.790571 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:11:00.790592 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:11:00.790637 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:11:00.794706 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:00.795457 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:00.795466 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:00.795470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:00.795667 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:00.796083 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:00.796088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:00.796092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:00.796598 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:11:00.796913 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:00.797272 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 01:11:00.797572 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:11:00.798771 21892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:11:00.922202 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 21769
I20260929 01:11:00.939446 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 21408
I20260929 01:11:00.940670 21470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:11:01.052356 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 21408
I20260929 01:11:21.073984 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:38197
--webserver_interface=127.5.198.254
--webserver_port=43013
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:38197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:11:21.406673 21946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:11:21.407013 21946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:11:21.407167 21946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:11:21.414574 21946 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 01:11:21.414713 21946 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 01:11:21.414806 21946 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 01:11:21.414902 21946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:11:21.415009 21946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:11:21.415128 21946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:11:21.415216 21946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:11:21.415297 21946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:11:21.425482 21946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:38197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:38197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:11:21.426826 21946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:11:21.428153 21946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:11:21.451799 21952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:21.451825 21953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:21.451735 21955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:11:21.452590 21946 server_base.cc:1034] running on GCE node
I20260929 01:11:21.453552 21946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:11:21.455241 21946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:11:21.456552 21946 hybrid_clock.cc:648] HybridClock initialized: now 1790644281456429 us; error 49 us; skew 500 ppm
I20260929 01:11:21.456975 21946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:11:21 dist-test-slave-41cl krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644281, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:11:21.466832 21946 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:11:21.468590 21946 webserver.cc:460] Webserver started at http://127.5.198.254:43013/ using document root <none> and password file <none>
I20260929 01:11:21.469388 21946 fs_manager.cc:359] Metadata directory not provided
I20260929 01:11:21.469542 21946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:11:21.475613 21946 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:11:21.478786 21962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:21.480263 21946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:11:21.480542 21946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "3f4f2894592c4439b04146fe0419db90"
format_stamp: "Formatted at 2026-09-29 01:10:58 on dist-test-slave-41cl"
server_key: "f4eb7aadaddcdf67f10003e09c2add67"
server_key_iv: "922cd4e9b098a4174f45183176ab8515"
server_key_version: "encryptionkey@0"
I20260929 01:11:21.481658 21946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:11:21.499420 21946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:11:21.504240 21946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:11:21.504630 21946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:11:21.525092 21946 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:38197
I20260929 01:11:21.525141 22013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:38197 every 8 connection(s)
I20260929 01:11:21.526777 21946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:11:21.527500 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 21946
I20260929 01:11:21.528574 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:37583
--local_ip_for_outbound_sockets=127.5.198.193
--tserver_master_addrs=127.5.198.254:38197
--webserver_port=36061
--webserver_interface=127.5.198.193
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:11:21.534590 22014 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:11:21.538239 22014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Bootstrap starting.
I20260929 01:11:21.559576 22014 log.cc:824] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Log is configured to *not* fsync() on all Append() calls
I20260929 01:11:21.572369 22014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:11:21.573007 22014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Bootstrap complete.
I20260929 01:11:21.582813 22014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } }
I20260929 01:11:21.583783 22014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4f2894592c4439b04146fe0419db90, State: Initialized, Role: FOLLOWER
I20260929 01:11:21.584573 22014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } }
I20260929 01:11:21.584852 22014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:11:21.584990 22014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:11:21.585197 22014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:11:21.589529 22014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } }
I20260929 01:11:21.590075 22014 leader_election.cc:302] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f4f2894592c4439b04146fe0419db90; no voters:
I20260929 01:11:21.590817 22014 leader_election.cc:288] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 01:11:21.590976 22018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:11:21.592269 22018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 2 LEADER]: Becoming Leader. State: Replica: 3f4f2894592c4439b04146fe0419db90, State: Running, Role: LEADER
I20260929 01:11:21.592979 22018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } }
I20260929 01:11:21.593377 22014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:11:21.595623 22020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f4f2894592c4439b04146fe0419db90. Latest consensus state: current_term: 2 leader_uuid: "3f4f2894592c4439b04146fe0419db90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } } }
I20260929 01:11:21.596122 22020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [sys.catalog]: This master's current role is: LEADER
I20260929 01:11:21.596793 22019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3f4f2894592c4439b04146fe0419db90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } } }
I20260929 01:11:21.597193 22019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [sys.catalog]: This master's current role is: LEADER
I20260929 01:11:21.606066 22024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:11:21.617317 22024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:11:21.618772 22024 catalog_manager.cc:1259] Loaded cluster ID: 8d463c170fea4ff0a16634f7f6ee261c
I20260929 01:11:21.618947 22024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:11:21.623108 22024 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:11:21.624768 22024 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Loaded TSK: 0
I20260929 01:11:21.657099 22024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Generated new TSK 1
I20260929 01:11:21.657595 22024 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 01:11:21.886349 22016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:11:21.886667 22016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:11:21.886828 22016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:11:21.894330 22016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:11:21.894472 22016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:11:21.894681 22016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:11:21.904757 22016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:37583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=36061
--tserver_master_addrs=127.5.198.254:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:11:21.906105 22016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:11:21.907428 22016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:11:21.922693 22043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:21.922719 22042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:21.923370 22045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:11:21.923940 22016 server_base.cc:1034] running on GCE node
I20260929 01:11:21.924752 22016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:11:21.926110 22016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:11:21.927371 22016 hybrid_clock.cc:648] HybridClock initialized: now 1790644281927320 us; error 36 us; skew 500 ppm
I20260929 01:11:21.927772 22016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:11:21 dist-test-slave-41cl krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644281, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:11:21.931695 22016 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:11:21.933316 22016 webserver.cc:460] Webserver started at http://127.5.198.193:36061/ using document root <none> and password file <none>
I20260929 01:11:21.934018 22016 fs_manager.cc:359] Metadata directory not provided
I20260929 01:11:21.934177 22016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:11:21.939756 22016 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:11:21.942968 22052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:21.944636 22016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:11:21.944821 22016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "abff705fff9346c9a09fb1dc9c46af69"
format_stamp: "Formatted at 2026-09-29 01:10:59 on dist-test-slave-41cl"
server_key: "ff97f94e0a3a05b1fd0d49afed271a54"
server_key_iv: "5dc21860773cf0ab4766e4e2a3dfa77b"
server_key_version: "encryptionkey@0"
I20260929 01:11:21.945863 22016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:11:21.974637 22016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:11:21.979323 22016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:11:21.979610 22016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:11:21.980683 22016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:11:21.982175 22016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:11:21.982302 22016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:21.982434 22016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:11:21.982519 22016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:22.016621 22016 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:37583
I20260929 01:11:22.016705 22164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:37583 every 8 connection(s)
I20260929 01:11:22.018203 22016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:11:22.022243 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 22016
I20260929 01:11:22.023063 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:40439
--local_ip_for_outbound_sockets=127.5.198.194
--tserver_master_addrs=127.5.198.254:38197
--webserver_port=41895
--webserver_interface=127.5.198.194
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:11:22 dist-test-slave-41cl krb5kdc[21395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644281, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:11:22.045442 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:22.021502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:55769)
0929 01:11:22.022438 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:22.022507 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 01:11:22.025299 (+ 2792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:11:22.026727 (+ 1428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:11:22.026739 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:11:22.027521 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:11:22.028320 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:22.028346 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:22.030032 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:22.030041 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:22.031477 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:22.031489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:22.033008 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:22.033013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:22.033192 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:22.033891 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:11:22.033912 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:11:22.036261 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:11:22.041027 (+ 4766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:11:22.041034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:11:22.041045 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:11:22.041440 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:11:22.041985 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:11:22.041989 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:11:22.041991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:11:22.042201 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:11:22.043195 (+ 994us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:11:22.043201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:11:22.043577 (+ 376us) client_negotiation.cc:769] Sending connection context
0929 01:11:22.043817 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 01:11:22.044248 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":360,"threads_started":1}
I20260929 01:11:22.046054 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:22.021179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:55769 (local address 127.5.198.254:38197)
0929 01:11:22.021919 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:22.021946 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:22.022656 (+ 710us) server_negotiation.cc:407] Connection header received
0929 01:11:22.025869 (+ 3213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:22.025881 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:22.026220 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:22.026527 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:11:22.028592 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:22.029850 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:22.031638 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:22.032787 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:22.032997 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:22.036604 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:11:22.036617 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:11:22.036640 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:11:22.036690 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:11:22.040877 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:22.041594 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:22.041606 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:22.041617 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:22.041830 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:22.042328 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:22.042335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:22.042342 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:22.043036 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:11:22.043254 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:22.044100 (+ 846us) server_negotiation.cc:299] Negotiation successful
0929 01:11:22.044631 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":257,"threads_started":1}
I20260929 01:11:22.049013 22165 heartbeater.cc:347] Connected to a master server at 127.5.198.254:38197
I20260929 01:11:22.049329 22165 heartbeater.cc:464] Registering TS with master...
I20260929 01:11:22.050107 22165 heartbeater.cc:511] Master 127.5.198.254:38197 requested a full tablet report, sending...
I20260929 01:11:22.052297 21979 ts_manager.cc:194] Registered new tserver with Master: abff705fff9346c9a09fb1dc9c46af69 (127.5.198.193:37583)
I20260929 01:11:22.053879 21979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:55769
W20260929 01:11:22.366209 22169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:11:22.366539 22169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:11:22.366730 22169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:11:22.373974 22169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:11:22.374135 22169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:11:22.374336 22169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:11:22.384027 22169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:40439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=41895
--tserver_master_addrs=127.5.198.254:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:11:22.385377 22169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:11:22.386760 22169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:11:22.400075 22175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:22.400293 22174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:22.400527 22177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:11:22.401445 22169 server_base.cc:1034] running on GCE node
I20260929 01:11:22.402083 22169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:11:22.402998 22169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:11:22.404224 22169 hybrid_clock.cc:648] HybridClock initialized: now 1790644282404156 us; error 50 us; skew 500 ppm
I20260929 01:11:22.404618 22169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:11:22 dist-test-slave-41cl krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644282, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:11:22.407843 22169 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:11:22.409451 22169 webserver.cc:460] Webserver started at http://127.5.198.194:41895/ using document root <none> and password file <none>
I20260929 01:11:22.410116 22169 fs_manager.cc:359] Metadata directory not provided
I20260929 01:11:22.410267 22169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:11:22.416047 22169 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:11:22.419240 22184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:22.420662 22169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:11:22.420841 22169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "ef8ca4430404475a90195e362b3bae28"
format_stamp: "Formatted at 2026-09-29 01:10:59 on dist-test-slave-41cl"
server_key: "98f98ec1fb99500ef655dccfa8b4c636"
server_key_iv: "10d2d537ea6fae3f7e2623cfb1162b06"
server_key_version: "encryptionkey@0"
I20260929 01:11:22.421793 22169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:11:22.458720 22169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:11:22.463227 22169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:11:22.463554 22169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:11:22.464627 22169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:11:22.465997 22169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:11:22.466094 22169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:22.466253 22169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:11:22.466346 22169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:22.500128 22169 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:40439
I20260929 01:11:22.500217 22296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:40439 every 8 connection(s)
I20260929 01:11:22.501611 22169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:11:22.511104 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 22169
I20260929 01:11:22.511833 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:45563
--local_ip_for_outbound_sockets=127.5.198.195
--tserver_master_addrs=127.5.198.254:38197
--webserver_port=36103
--webserver_interface=127.5.198.195
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:11:22 dist-test-slave-41cl krb5kdc[21395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644282, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:11:22.523273 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:22.504329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:47287 (local address 127.5.198.254:38197)
0929 01:11:22.504569 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:22.504575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:22.505750 (+ 1175us) server_negotiation.cc:407] Connection header received
0929 01:11:22.507646 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:22.507650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:22.507721 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:22.507810 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:11:22.509372 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:22.510113 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:22.512304 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:22.512789 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:22.512891 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:22.517285 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:11:22.517290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:11:22.517293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:11:22.517327 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:11:22.520270 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:22.521036 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:22.521041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:22.521050 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:22.521165 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:22.521588 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:22.521593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:22.521596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:22.521876 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:11:22.522035 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:22.522702 (+ 667us) server_negotiation.cc:299] Negotiation successful
0929 01:11:22.522939 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 01:11:22.524336 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:22.504874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:47287)
0929 01:11:22.505589 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:22.505625 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 01:11:22.507351 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:11:22.508049 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:11:22.508057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:11:22.508669 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:11:22.509214 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:22.509229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:22.510273 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:22.510277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:22.512128 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:22.512145 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:22.513025 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:22.513035 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:22.513361 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:22.514149 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:11:22.514189 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:11:22.517045 (+ 2856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:11:22.520416 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:11:22.520424 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:11:22.520441 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:11:22.520899 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:11:22.521276 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:11:22.521281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:11:22.521285 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:11:22.521469 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:11:22.522009 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:11:22.522017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:11:22.522352 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 01:11:22.522609 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 01:11:22.523155 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":275,"threads_started":1}
I20260929 01:11:22.526201 22297 heartbeater.cc:347] Connected to a master server at 127.5.198.254:38197
I20260929 01:11:22.526539 22297 heartbeater.cc:464] Registering TS with master...
I20260929 01:11:22.527360 22297 heartbeater.cc:511] Master 127.5.198.254:38197 requested a full tablet report, sending...
I20260929 01:11:22.529001 21979 ts_manager.cc:194] Registered new tserver with Master: ef8ca4430404475a90195e362b3bae28 (127.5.198.194:40439)
I20260929 01:11:22.530309 21979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:47287
W20260929 01:11:22.838590 22300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:11:22.838842 22300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:11:22.838994 22300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:11:22.846174 22300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:11:22.846295 22300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:11:22.846477 22300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:11:22.856675 22300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:45563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=36103
--tserver_master_addrs=127.5.198.254:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:11:22.857879 22300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:11:22.859198 22300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:11:22.872329 22305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:22.872385 22308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:22.872308 22306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:11:22.873144 22300 server_base.cc:1034] running on GCE node
I20260929 01:11:22.873730 22300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:11:22.874650 22300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:11:22.875883 22300 hybrid_clock.cc:648] HybridClock initialized: now 1790644282875841 us; error 55 us; skew 500 ppm
I20260929 01:11:22.876240 22300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:11:22 dist-test-slave-41cl krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644282, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:11:22.879593 22300 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:11:22.881147 22300 webserver.cc:460] Webserver started at http://127.5.198.195:36103/ using document root <none> and password file <none>
I20260929 01:11:22.881840 22300 fs_manager.cc:359] Metadata directory not provided
I20260929 01:11:22.882000 22300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:11:22.887553 22300 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:11:22.890271 22315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:22.891554 22300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:11:22.891721 22300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "80f5a55a3a2442dab20236503cef7729"
format_stamp: "Formatted at 2026-09-29 01:11:00 on dist-test-slave-41cl"
server_key: "064fd0f3193fe7d19769e3294bdf7ebd"
server_key_iv: "d5475b670a9da224a8df2fa0d0c38b36"
server_key_version: "encryptionkey@0"
I20260929 01:11:22.892740 22300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:11:22.908215 22300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:11:22.912189 22300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:11:22.912492 22300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:11:22.913544 22300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:11:22.914897 22300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:11:22.915019 22300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:22.915155 22300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:11:22.915248 22300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:22.949211 22300 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:45563
I20260929 01:11:22.949321 22427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:45563 every 8 connection(s)
I20260929 01:11:22.950773 22300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:11:22.957563 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 22300
Sep 29 01:11:22 dist-test-slave-41cl krb5kdc[21395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644282, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:11:22.972340 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:22.953358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:46049 (local address 127.5.198.254:38197)
0929 01:11:22.953584 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:22.953591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:22.954680 (+ 1089us) server_negotiation.cc:407] Connection header received
0929 01:11:22.956430 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:22.956435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:22.956556 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:22.956655 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:11:22.958195 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:22.959177 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:22.960733 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:22.961170 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:22.961311 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:22.965306 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:11:22.965313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:11:22.965318 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:11:22.965358 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:11:22.968785 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:22.969719 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:22.969724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:22.969727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:22.969888 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:22.970431 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:22.970438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:22.970442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:22.970782 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:11:22.970902 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:22.971795 (+ 893us) server_negotiation.cc:299] Negotiation successful
0929 01:11:22.972009 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 01:11:22.973677 22430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:22.953847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:46049)
0929 01:11:22.954522 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:22.954558 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 01:11:22.956181 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:11:22.956851 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:11:22.956858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:11:22.957399 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:11:22.957999 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:22.958023 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:22.959337 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:22.959344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:22.960587 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:22.960599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:22.961400 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:22.961408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:22.961657 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:22.962396 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:11:22.962419 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:11:22.965090 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:11:22.968928 (+ 3838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:11:22.968937 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:11:22.968954 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:11:22.969575 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:11:22.970021 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:11:22.970025 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:11:22.970029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:11:22.970283 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:11:22.970917 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:11:22.970927 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:11:22.971384 (+ 457us) client_negotiation.cc:769] Sending connection context
0929 01:11:22.971719 (+ 335us) client_negotiation.cc:240] Negotiation successful
0929 01:11:22.972266 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":270,"threads_started":1}
I20260929 01:11:22.975061 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:22.958489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.5.198.254:38197)
0929 01:11:22.959081 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:22.959089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:22.959615 (+ 526us) server_negotiation.cc:407] Connection header received
0929 01:11:22.959803 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:22.959808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:22.959879 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:22.959987 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:11:22.961584 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:22.962523 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:22.964307 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:22.964983 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:22.965128 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:22.968699 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:11:22.968703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:11:22.968706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:11:22.968752 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:11:22.972100 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:22.972906 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:22.972911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:22.972915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:22.973052 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:22.973413 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:22.973418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:22.973421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:22.973752 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:11:22.974125 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:22.974459 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 01:11:22.974685 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":248,"threads_started":1}
I20260929 01:11:22.975478 22428 heartbeater.cc:347] Connected to a master server at 127.5.198.254:38197
I20260929 01:11:22.975805 22428 heartbeater.cc:464] Registering TS with master...
I20260929 01:11:22.976585 22428 heartbeater.cc:511] Master 127.5.198.254:38197 requested a full tablet report, sending...
I20260929 01:11:22.978097 21978 ts_manager.cc:194] Registered new tserver with Master: 80f5a55a3a2442dab20236503cef7729 (127.5.198.195:45563)
I20260929 01:11:22.978948 21978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:46049
I20260929 01:11:22.990810 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:11:22.998420 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:22.991862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38272 (local address 127.5.198.254:38197)
0929 01:11:22.992047 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:22.992052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:22.992653 (+ 601us) server_negotiation.cc:407] Connection header received
0929 01:11:22.992890 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:22.992894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:22.993063 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:22.993201 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:11:22.993727 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:22.994434 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:22.996366 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:22.996966 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:22.997070 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:22.997681 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:11:22.997883 (+ 202us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 01:11:22.998185 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38272: token expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 01:11:22.998528 22431 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38272: token expired
W20260929 01:11:22.999176 21901 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.198.254:38197)
I20260929 01:11:23.012168 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:22.999644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38288 (local address 127.5.198.254:38197)
0929 01:11:22.999791 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:22.999795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:22.999843 (+ 48us) server_negotiation.cc:407] Connection header received
0929 01:11:22.999959 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:22.999963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:23.000019 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:23.000100 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:11:23.001497 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.002175 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.003477 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.003865 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.003959 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:23.006856 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:11:23.006860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:11:23.006864 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:11:23.006916 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:11:23.009686 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:23.010369 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:23.010374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:23.010378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:23.010517 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:23.010940 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:23.010945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:23.010950 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:23.011311 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:11:23.011526 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:23.011777 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 01:11:23.011932 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:11:23.015635 21978 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:11:23.023001 21978 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:11:23.047138 22439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.034440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.5.198.193:37583)
0929 01:11:23.039074 (+ 4634us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:23.039081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:23.039103 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:11:23.039172 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:23.039177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:23.039397 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:23.039598 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:11:23.040259 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.041813 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.044422 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.045905 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.046064 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:23.046309 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:23.046638 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 01:11:23.046797 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4491,"thread_start_us":4351,"threads_started":1}
I20260929 01:11:23.047210 22438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.034252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37583 (local address 127.0.0.1:47942)
0929 01:11:23.035319 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:23.035358 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:11:23.035528 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:11:23.039606 (+ 4078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:11:23.039610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:11:23.039643 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:11:23.040064 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:23.040076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.041933 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.041940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:23.044262 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:23.044276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.046147 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.046153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:23.046389 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:23.046452 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 01:11:23.046577 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 01:11:23.046796 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":197,"thread_start_us":1482,"threads_started":1}
I20260929 01:11:23.047392 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.035512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34852 (local address 127.5.198.194:40439)
0929 01:11:23.036909 (+ 1397us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:23.036922 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:23.038073 (+ 1151us) server_negotiation.cc:407] Connection header received
0929 01:11:23.038252 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:23.038263 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:23.038536 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:23.039464 (+ 928us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:11:23.039683 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.041514 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.044979 (+ 3465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.045939 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.046102 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:23.046289 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:23.046957 (+ 668us) server_negotiation.cc:299] Negotiation successful
0929 01:11:23.047104 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1209,"thread_start_us":1115,"threads_started":1}
I20260929 01:11:23.047999 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.034768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:45563 (local address 127.0.0.1:39126)
0929 01:11:23.037357 (+ 2589us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:23.037409 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 01:11:23.037514 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:11:23.039275 (+ 1761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:11:23.039280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:11:23.039309 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:11:23.039680 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:23.039698 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.041649 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.041657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:23.044603 (+ 2946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:23.044614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.046358 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.046365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:23.046498 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:23.046520 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:11:23.046619 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 01:11:23.047667 (+ 1048us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2469,"mutex_wait_us":1108,"thread_start_us":2955,"threads_started":1}
I20260929 01:11:23.048185 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.035637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39126 (local address 127.5.198.195:45563)
0929 01:11:23.038692 (+ 3055us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:23.038703 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:23.038736 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:11:23.038828 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:23.038834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:23.039072 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:23.039279 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:11:23.039935 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.041515 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.044747 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.046111 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.046744 (+ 633us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:23.046941 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:23.047010 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 01:11:23.047161 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2876,"thread_start_us":2712,"threads_started":1}
I20260929 01:11:23.048662 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.034977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:40439 (local address 127.0.0.1:34852)
0929 01:11:23.037948 (+ 2971us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:23.037966 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:11:23.038109 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:11:23.038777 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:11:23.038784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:11:23.038827 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:11:23.039245 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:23.039259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.041674 (+ 2415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.041679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:23.044846 (+ 3167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:23.044854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.046452 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.046457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:23.046625 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:23.046660 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 01:11:23.046734 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 01:11:23.046828 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2875,"mutex_wait_us":248,"thread_start_us":2132,"threads_started":1}
I20260929 01:11:23.051770 22100 tablet_service.cc:1467] Processing CreateTablet for tablet c77ad59ebc6345148c6327725c0aacd5 (DEFAULT_TABLE table=test-table [id=2bbe3a1f0281455bae2e7702f567bc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:11:23.051898 22232 tablet_service.cc:1467] Processing CreateTablet for tablet c77ad59ebc6345148c6327725c0aacd5 (DEFAULT_TABLE table=test-table [id=2bbe3a1f0281455bae2e7702f567bc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:11:23.051896 22363 tablet_service.cc:1467] Processing CreateTablet for tablet c77ad59ebc6345148c6327725c0aacd5 (DEFAULT_TABLE table=test-table [id=2bbe3a1f0281455bae2e7702f567bc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:11:23.053206 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77ad59ebc6345148c6327725c0aacd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:11:23.053231 22232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77ad59ebc6345148c6327725c0aacd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:11:23.053692 22363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77ad59ebc6345148c6327725c0aacd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:11:23.056713 22165 heartbeater.cc:503] Master 127.5.198.254:38197 was elected leader, sending a full tablet report...
I20260929 01:11:23.067073 22444 tablet_bootstrap.cc:492] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69: Bootstrap starting.
I20260929 01:11:23.068420 22445 tablet_bootstrap.cc:492] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28: Bootstrap starting.
I20260929 01:11:23.070915 22444 tablet_bootstrap.cc:654] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69: Neither blocks nor log segments found. Creating new log.
I20260929 01:11:23.071905 22445 tablet_bootstrap.cc:654] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28: Neither blocks nor log segments found. Creating new log.
I20260929 01:11:23.072229 22444 log.cc:824] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69: Log is configured to *not* fsync() on all Append() calls
I20260929 01:11:23.073203 22445 log.cc:824] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28: Log is configured to *not* fsync() on all Append() calls
I20260929 01:11:23.075551 22444 tablet_bootstrap.cc:492] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69: No bootstrap required, opened a new log
I20260929 01:11:23.075829 22444 ts_tablet_manager.cc:1397] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260929 01:11:23.076417 22445 tablet_bootstrap.cc:492] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28: No bootstrap required, opened a new log
I20260929 01:11:23.076722 22445 ts_tablet_manager.cc:1397] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260929 01:11:23.080550 22446 tablet_bootstrap.cc:492] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729: Bootstrap starting.
I20260929 01:11:23.083264 22446 tablet_bootstrap.cc:654] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729: Neither blocks nor log segments found. Creating new log.
I20260929 01:11:23.084227 22446 log.cc:824] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729: Log is configured to *not* fsync() on all Append() calls
I20260929 01:11:23.086645 22444 raft_consensus.cc:348] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:23.087110 22444 raft_consensus.cc:374] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:11:23.087217 22444 raft_consensus.cc:729] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abff705fff9346c9a09fb1dc9c46af69, State: Initialized, Role: FOLLOWER
I20260929 01:11:23.087056 22445 raft_consensus.cc:348] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:23.087387 22446 tablet_bootstrap.cc:492] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729: No bootstrap required, opened a new log
I20260929 01:11:23.087478 22445 raft_consensus.cc:374] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:11:23.087630 22445 raft_consensus.cc:729] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8ca4430404475a90195e362b3bae28, State: Initialized, Role: FOLLOWER
I20260929 01:11:23.087662 22446 ts_tablet_manager.cc:1397] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260929 01:11:23.087950 22444 consensus_queue.cc:260] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:23.088164 22445 consensus_queue.cc:260] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:23.089303 22297 heartbeater.cc:503] Master 127.5.198.254:38197 was elected leader, sending a full tablet report...
I20260929 01:11:23.089967 22445 ts_tablet_manager.cc:1428] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260929 01:11:23.091473 22444 ts_tablet_manager.cc:1428] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260929 01:11:23.096966 22446 raft_consensus.cc:348] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:23.097328 22446 raft_consensus.cc:374] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:11:23.097430 22446 raft_consensus.cc:729] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f5a55a3a2442dab20236503cef7729, State: Initialized, Role: FOLLOWER
I20260929 01:11:23.097960 22446 consensus_queue.cc:260] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:23.098881 22428 heartbeater.cc:503] Master 127.5.198.254:38197 was elected leader, sending a full tablet report...
I20260929 01:11:23.099469 22446 ts_tablet_manager.cc:1428] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260929 01:11:23.157852 22451 raft_consensus.cc:482] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:11:23.158195 22451 raft_consensus.cc:504] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:23.160133 22451 leader_election.cc:288] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abff705fff9346c9a09fb1dc9c46af69 (127.5.198.193:37583), 80f5a55a3a2442dab20236503cef7729 (127.5.198.195:45563)
I20260929 01:11:23.167080 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.161278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:46071 (local address 127.5.198.195:45563)
0929 01:11:23.161463 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:23.161471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:23.161570 (+ 99us) server_negotiation.cc:407] Connection header received
0929 01:11:23.161791 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:23.161796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:23.161862 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:23.161960 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:11:23.162588 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.163296 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.165120 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.165978 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.166160 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:23.166244 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:23.166685 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 01:11:23.166806 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 01:11:23.167215 22439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.160695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:46381 (local address 127.5.198.193:37583)
0929 01:11:23.160905 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:23.160912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:23.161298 (+ 386us) server_negotiation.cc:407] Connection header received
0929 01:11:23.161606 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:23.161610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:23.161674 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:23.161798 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:11:23.162418 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.163150 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.165125 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.165954 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.166148 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:23.166232 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:23.166742 (+ 510us) server_negotiation.cc:299] Negotiation successful
0929 01:11:23.166941 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 01:11:23.167297 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.160306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37583 (local address 127.5.198.194:46381)
0929 01:11:23.161187 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:23.161207 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:11:23.161433 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:11:23.161825 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:11:23.161829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:11:23.161854 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:11:23.162260 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:23.162273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.163295 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.163301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:23.164972 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:23.164984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.166202 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.166209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:23.166541 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:23.166574 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:11:23.166687 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 01:11:23.166813 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":92,"spinlock_wait_cycles":16128,"thread_start_us":453,"threads_started":1}
I20260929 01:11:23.167384 22454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.160617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:45563 (local address 127.5.198.194:46071)
0929 01:11:23.161456 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:23.161475 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:11:23.161656 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:11:23.161980 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:11:23.161984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:11:23.162023 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:11:23.162430 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:23.162438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.163434 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.163439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:23.164968 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:23.164980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.166241 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.166247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:23.166407 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:23.166547 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 01:11:23.166650 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 01:11:23.166774 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":96,"thread_start_us":469,"threads_started":1}
I20260929 01:11:23.168366 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77ad59ebc6345148c6327725c0aacd5" candidate_uuid: "ef8ca4430404475a90195e362b3bae28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abff705fff9346c9a09fb1dc9c46af69" is_pre_election: true
I20260929 01:11:23.168637 22383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77ad59ebc6345148c6327725c0aacd5" candidate_uuid: "ef8ca4430404475a90195e362b3bae28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f5a55a3a2442dab20236503cef7729" is_pre_election: true
I20260929 01:11:23.168965 22120 raft_consensus.cc:2454] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef8ca4430404475a90195e362b3bae28 in term 0.
I20260929 01:11:23.169142 22383 raft_consensus.cc:2454] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef8ca4430404475a90195e362b3bae28 in term 0.
I20260929 01:11:23.169658 22186 leader_election.cc:302] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 80f5a55a3a2442dab20236503cef7729, abff705fff9346c9a09fb1dc9c46af69, ef8ca4430404475a90195e362b3bae28; no voters:
I20260929 01:11:23.170082 22451 raft_consensus.cc:2781] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:11:23.170223 22451 raft_consensus.cc:482] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:11:23.170336 22451 raft_consensus.cc:3037] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:11:23.172592 22451 raft_consensus.cc:504] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:23.173174 22451 leader_election.cc:288] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [CANDIDATE]: Term 1 election: Requested vote from peers abff705fff9346c9a09fb1dc9c46af69 (127.5.198.193:37583), 80f5a55a3a2442dab20236503cef7729 (127.5.198.195:45563)
I20260929 01:11:23.173593 22120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77ad59ebc6345148c6327725c0aacd5" candidate_uuid: "ef8ca4430404475a90195e362b3bae28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abff705fff9346c9a09fb1dc9c46af69"
I20260929 01:11:23.173597 22383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77ad59ebc6345148c6327725c0aacd5" candidate_uuid: "ef8ca4430404475a90195e362b3bae28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f5a55a3a2442dab20236503cef7729"
I20260929 01:11:23.173770 22120 raft_consensus.cc:3037] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:11:23.173789 22383 raft_consensus.cc:3037] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:11:23.175849 22120 raft_consensus.cc:2454] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef8ca4430404475a90195e362b3bae28 in term 1.
I20260929 01:11:23.176049 22383 raft_consensus.cc:2454] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef8ca4430404475a90195e362b3bae28 in term 1.
I20260929 01:11:23.176260 22186 leader_election.cc:302] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abff705fff9346c9a09fb1dc9c46af69, ef8ca4430404475a90195e362b3bae28; no voters:
I20260929 01:11:23.176580 22451 raft_consensus.cc:2781] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:11:23.177053 22451 raft_consensus.cc:686] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [term 1 LEADER]: Becoming Leader. State: Replica: ef8ca4430404475a90195e362b3bae28, State: Running, Role: LEADER
I20260929 01:11:23.177502 22451 consensus_queue.cc:237] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:23.182089 21978 catalog_manager.cc:5495] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef8ca4430404475a90195e362b3bae28 (127.5.198.194). New cstate: current_term: 1 leader_uuid: "ef8ca4430404475a90195e362b3bae28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } health_report { overall_health: HEALTHY } } }
I20260929 01:11:23.201969 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.196644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34860 (local address 127.5.198.194:40439)
0929 01:11:23.196851 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:23.196856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:23.196872 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:11:23.197151 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:23.197155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:23.197238 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:23.197381 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:11:23.197742 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.198453 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.199864 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.200268 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.200396 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:23.200894 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:11:23.201366 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:11:23.201530 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:23.201636 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 01:11:23.201715 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 01:11:23.202340 22429 tablet.cc:2369] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:11:23.210237 22383 raft_consensus.cc:1261] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729 [term 1 FOLLOWER]: Refusing update from remote peer ef8ca4430404475a90195e362b3bae28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:11:23.210237 22120 raft_consensus.cc:1261] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Refusing update from remote peer ef8ca4430404475a90195e362b3bae28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:11:23.211325 22455 consensus_queue.cc:1035] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:11:23.211706 22451 consensus_queue.cc:1035] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:11:23.239590 21978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:11:23.239848 21978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:11:23.243268 21978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Sending DeleteTablet for 3 replicas of tablet c77ad59ebc6345148c6327725c0aacd5
I20260929 01:11:23.244573 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet c77ad59ebc6345148c6327725c0aacd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:11:23 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:47942
I20260929 01:11:23.244660 22363 tablet_service.cc:1514] Processing DeleteTablet for tablet c77ad59ebc6345148c6327725c0aacd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:11:23 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:39126
I20260929 01:11:23.244874 22232 tablet_service.cc:1514] Processing DeleteTablet for tablet c77ad59ebc6345148c6327725c0aacd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:11:23 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:34852
I20260929 01:11:23.246027 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 22016
I20260929 01:11:23.257721 22470 tablet_replica.cc:330] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28: stopping tablet replica
I20260929 01:11:23.258885 22470 raft_consensus.cc:2229] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:11:23.260713 22470 raft_consensus.cc:2258] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:11:23.264173 22470 ts_tablet_manager.cc:1905] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:11:23.269033 22469 tablet_replica.cc:330] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729: stopping tablet replica
I20260929 01:11:23.269418 22439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.246916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47948 (local address 127.5.198.193:37583)
0929 01:11:23.247101 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:23.247107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:23.247209 (+ 102us) server_negotiation.cc:407] Connection header received
0929 01:11:23.247408 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:23.247416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:23.247514 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:23.247628 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:11:23.250759 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.252339 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.255020 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.255980 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.256144 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:23.260443 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:11:23.260461 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:11:23.260500 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:11:23.260547 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:11:23.264869 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:23.265953 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:23.265961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:23.265964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:23.266136 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:23.266618 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:23.266623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:23.266627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:23.267094 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:11:23.267397 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:23.268813 (+ 1416us) server_negotiation.cc:299] Negotiation successful
0929 01:11:23.269078 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 01:11:23.269479 22469 raft_consensus.cc:2229] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:11:23.270010 22469 raft_consensus.cc:2258] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:11:23.270131 22160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:11:23.274044 22469 ts_tablet_manager.cc:1905] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:11:23.277719 22470 ts_tablet_manager.cc:1918] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:11:23.277989 22470 log.cc:1196] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/c77ad59ebc6345148c6327725c0aacd5
I20260929 01:11:23.278616 22470 ts_tablet_manager.cc:1939] T c77ad59ebc6345148c6327725c0aacd5 P ef8ca4430404475a90195e362b3bae28: Deleting consensus metadata
I20260929 01:11:23.280552 22469 ts_tablet_manager.cc:1918] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:11:23.280611 21963 catalog_manager.cc:4841] TS ef8ca4430404475a90195e362b3bae28 (127.5.198.194:40439): tablet c77ad59ebc6345148c6327725c0aacd5 (table test-table [id=2bbe3a1f0281455bae2e7702f567bc70]) successfully deleted
I20260929 01:11:23.280730 22469 log.cc:1196] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/c77ad59ebc6345148c6327725c0aacd5
I20260929 01:11:23.281181 22469 ts_tablet_manager.cc:1939] T c77ad59ebc6345148c6327725c0aacd5 P 80f5a55a3a2442dab20236503cef7729: Deleting consensus metadata
I20260929 01:11:23.282603 21966 catalog_manager.cc:4841] TS 80f5a55a3a2442dab20236503cef7729 (127.5.198.195:45563): tablet c77ad59ebc6345148c6327725c0aacd5 (table test-table [id=2bbe3a1f0281455bae2e7702f567bc70]) successfully deleted
I20260929 01:11:23.381700 22468 tablet_replica.cc:330] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69: stopping tablet replica
I20260929 01:11:23.382467 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 22016
W20260929 01:11:23.401432 21964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.198.193:37583 (error 108)
I20260929 01:11:23.401667 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 22169
I20260929 01:11:23.403594 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.403130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37583 (local address 127.0.0.1:34746)
0929 01:11:23.403314 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:23.403432 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 01:11:23.403827 21964 catalog_manager.cc:4584] TS abff705fff9346c9a09fb1dc9c46af69 (127.5.198.193:37583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c77ad59ebc6345148c6327725c0aacd5: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37583: connect: Connection refused (error 111)
I20260929 01:11:23.417273 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.402637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.5.198.194:40439)
0929 01:11:23.402846 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:23.402850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:23.402971 (+ 121us) server_negotiation.cc:407] Connection header received
0929 01:11:23.403181 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:23.403186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:23.403278 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:23.403419 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:11:23.404952 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.405771 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.407193 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.407706 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.407860 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:23.410619 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:11:23.410632 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:11:23.410668 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:11:23.410724 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:11:23.414071 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:23.414893 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:23.414915 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:23.414918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:23.415068 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:23.415495 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:23.415501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:23.415504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:23.415890 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:11:23.416253 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:23.416633 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 01:11:23.416867 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 01:11:23.418128 22292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:11:23.457616 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.457070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37583 (local address 127.0.0.1:34762)
0929 01:11:23.457302 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:23.457413 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 01:11:23.458768 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.458378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37583 (local address 127.0.0.1:34766)
0929 01:11:23.458546 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:23.458645 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 01:11:23.519553 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.519047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37583 (local address 127.0.0.1:34780)
0929 01:11:23.519264 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:23.519373 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 01:11:23.520639 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.520216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37583 (local address 127.0.0.1:34782)
0929 01:11:23.520371 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:23.520505 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 01:11:23.533069 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 22169
I20260929 01:11:23.555493 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 22300
I20260929 01:11:23.568254 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.556180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.5.198.195:45563)
0929 01:11:23.556336 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:23.556340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:23.556356 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:11:23.556536 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:23.556540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:23.556620 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:23.556693 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:11:23.557932 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.558643 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.559933 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:23.560305 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:23.560393 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:23.563033 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:11:23.563054 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:11:23.563069 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:11:23.563104 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:11:23.565872 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:23.566573 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:23.566580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:23.566582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:23.566710 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:23.567039 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:23.567049 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:23.567051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:23.567379 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:11:23.567594 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:23.567832 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 01:11:23.567983 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 01:11:23.568920 22423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:11:23.601387 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.600856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37583 (local address 127.0.0.1:34796)
0929 01:11:23.601132 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:23.601224 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260929 01:11:23.602519 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:23.602070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37583 (local address 127.0.0.1:34808)
0929 01:11:23.602290 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:23.602362 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 01:11:23.659471 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 22300
I20260929 01:11:23.675935 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 21946
I20260929 01:11:23.677104 22009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:11:23.793694 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 21946
I20260929 01:11:43.815778 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:38197
--webserver_interface=127.5.198.254
--webserver_port=43013
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:38197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:11:44.150889 22476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:11:44.151202 22476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:11:44.151356 22476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:11:44.158484 22476 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 01:11:44.158607 22476 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 01:11:44.158694 22476 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 01:11:44.158778 22476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:11:44.158860 22476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:11:44.158982 22476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:11:44.159065 22476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:11:44.159144 22476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:11:44.168692 22476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:38197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:38197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:11:44.169960 22476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:11:44.171301 22476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:11:44.184906 22482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:44.184952 22484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:11:44.185109 22476 server_base.cc:1034] running on GCE node
W20260929 01:11:44.184859 22481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:11:44.186179 22476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:11:44.187615 22476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:11:44.188884 22476 hybrid_clock.cc:648] HybridClock initialized: now 1790644304188808 us; error 34 us; skew 500 ppm
I20260929 01:11:44.189239 22476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:11:44 dist-test-slave-41cl krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644304, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:11:44.199163 22476 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:11:44.200690 22476 webserver.cc:460] Webserver started at http://127.5.198.254:43013/ using document root <none> and password file <none>
I20260929 01:11:44.201407 22476 fs_manager.cc:359] Metadata directory not provided
I20260929 01:11:44.201535 22476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:11:44.206970 22476 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 01:11:44.210192 22491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:44.211486 22476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:11:44.211670 22476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "3f4f2894592c4439b04146fe0419db90"
format_stamp: "Formatted at 2026-09-29 01:10:58 on dist-test-slave-41cl"
server_key: "f4eb7aadaddcdf67f10003e09c2add67"
server_key_iv: "922cd4e9b098a4174f45183176ab8515"
server_key_version: "encryptionkey@0"
I20260929 01:11:44.213029 22476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:11:44.265215 22476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:11:44.269590 22476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:11:44.269927 22476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:11:44.288981 22476 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:38197
I20260929 01:11:44.289021 22542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:38197 every 8 connection(s)
I20260929 01:11:44.290524 22476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:11:44.297756 22543 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:11:44.300889 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 22476
I20260929 01:11:44.301098 22543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Bootstrap starting.
I20260929 01:11:44.301801 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:37583
--local_ip_for_outbound_sockets=127.5.198.193
--tserver_master_addrs=127.5.198.254:38197
--webserver_port=36061
--webserver_interface=127.5.198.193
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:11:44.326962 22543 log.cc:824] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Log is configured to *not* fsync() on all Append() calls
I20260929 01:11:44.348207 22543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:11:44.348878 22543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Bootstrap complete.
I20260929 01:11:44.362341 22543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } }
I20260929 01:11:44.363801 22543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4f2894592c4439b04146fe0419db90, State: Initialized, Role: FOLLOWER
I20260929 01:11:44.364552 22543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } }
I20260929 01:11:44.364852 22543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:11:44.364972 22543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:11:44.365167 22543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:11:44.371414 22543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } }
I20260929 01:11:44.372104 22543 leader_election.cc:302] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f4f2894592c4439b04146fe0419db90; no voters:
I20260929 01:11:44.373011 22543 leader_election.cc:288] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 01:11:44.373138 22547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:11:44.374567 22547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [term 3 LEADER]: Becoming Leader. State: Replica: 3f4f2894592c4439b04146fe0419db90, State: Running, Role: LEADER
I20260929 01:11:44.375207 22543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:11:44.375129 22547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } }
I20260929 01:11:44.377899 22548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3f4f2894592c4439b04146fe0419db90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } } }
I20260929 01:11:44.378273 22548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [sys.catalog]: This master's current role is: LEADER
I20260929 01:11:44.377512 22549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f4f2894592c4439b04146fe0419db90. Latest consensus state: current_term: 3 leader_uuid: "3f4f2894592c4439b04146fe0419db90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4f2894592c4439b04146fe0419db90" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 38197 } } }
I20260929 01:11:44.378831 22549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90 [sys.catalog]: This master's current role is: LEADER
I20260929 01:11:44.380306 22554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:11:44.389861 22554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:11:44.391309 22554 catalog_manager.cc:1259] Loaded cluster ID: 8d463c170fea4ff0a16634f7f6ee261c
I20260929 01:11:44.391455 22554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:11:44.395435 22554 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:11:44.397056 22554 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Loaded TSK: 0
I20260929 01:11:44.397257 22554 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f4f2894592c4439b04146fe0419db90: Loaded TSK: 1
F20260929 01:11:44.398345 22554 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0e8ca0e9ad google::LogMessage::Fail() at ??:0
@ 0x7f0e8ca1188a google::LogMessage::SendToLog() at ??:0
@ 0x7f0e8ca0e4de google::LogMessage::Flush() at ??:0
@ 0x7f0e8ca0fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0e907727d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0e9076173e kudu::security::TokenSigner::ImportKeys() at ??:0
I20260929 01:11:44.418078 22565 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f0ea243d1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f0ea243c783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f0ea24c82a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f0ea24c80c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f0ea25125e9 std::function<>::operator()() at ??:0
@ 0x7f0ea2440cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0ea243ec36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0ea24c6610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0ea24c6439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0e8f65a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0e8f65d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0e8f65d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0e8f638eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0e929126db start_thread at ??:0
@ 0x7f0e8b8a861f clone at ??:0
W20260929 01:11:44.713276 22545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:11:44.713554 22545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:11:44.713719 22545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:11:44.721249 22545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:11:44.721359 22545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:11:44.721550 22545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:11:44.731415 22545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:37583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=36061
--tserver_master_addrs=127.5.198.254:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:11:44.732715 22545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:11:44.734040 22545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:11:44.747213 22591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:44.747262 22593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:44.747234 22590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:11:44.747946 22545 server_base.cc:1034] running on GCE node
I20260929 01:11:44.748454 22545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:11:44.749382 22545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:11:44.750578 22545 hybrid_clock.cc:648] HybridClock initialized: now 1790644304750539 us; error 27 us; skew 500 ppm
I20260929 01:11:44.750835 22545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:11:44 dist-test-slave-41cl krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644304, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:11:44.758877 22545 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:11:44.760183 22545 webserver.cc:460] Webserver started at http://127.5.198.193:36061/ using document root <none> and password file <none>
I20260929 01:11:44.760855 22545 fs_manager.cc:359] Metadata directory not provided
I20260929 01:11:44.760954 22545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:11:44.766191 22545 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:11:44.770596 22600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:44.772536 22545 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260929 01:11:44.772699 22545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "abff705fff9346c9a09fb1dc9c46af69"
format_stamp: "Formatted at 2026-09-29 01:10:59 on dist-test-slave-41cl"
server_key: "ff97f94e0a3a05b1fd0d49afed271a54"
server_key_iv: "5dc21860773cf0ab4766e4e2a3dfa77b"
server_key_version: "encryptionkey@0"
I20260929 01:11:44.774241 22545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:11:44.802878 22545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:11:44.808219 22545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:11:44.808547 22545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:11:44.809608 22545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:11:44.811550 22607 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 01:11:44.817356 22545 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 01:11:44.817461 22545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 01:11:44.817612 22545 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 01:11:44.820870 22545 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 01:11:44.820950 22545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260929 01:11:44.821368 22607 tablet_bootstrap.cc:492] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69: Bootstrap starting.
I20260929 01:11:44.848558 22607 log.cc:824] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69: Log is configured to *not* fsync() on all Append() calls
I20260929 01:11:44.858158 22545 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:37583
I20260929 01:11:44.858181 22714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:37583 every 8 connection(s)
I20260929 01:11:44.859395 22545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:11:44.860778 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 22545
I20260929 01:11:44.861517 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:40439
--local_ip_for_outbound_sockets=127.5.198.194
--tserver_master_addrs=127.5.198.254:38197
--webserver_port=41895
--webserver_interface=127.5.198.194
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:11:44.861872 22607 tablet_bootstrap.cc:492] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:11:44.862434 22607 tablet_bootstrap.cc:492] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69: Bootstrap complete.
I20260929 01:11:44.863999 22607 ts_tablet_manager.cc:1397] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69: Time spent bootstrapping tablet: real 0.043s user 0.030s sys 0.008s
I20260929 01:11:44.868788 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:44.866523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:46411)
0929 01:11:44.867422 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:44.867744 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":411,"threads_started":1}
W20260929 01:11:44.869396 22715 heartbeater.cc:644] Failed to heartbeat to 127.5.198.254:38197 (0 consecutive failures): Network error: Failed to ping master at 127.5.198.254:38197: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
I20260929 01:11:44.870306 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:44.869933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:49421)
0929 01:11:44.870072 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:44.870149 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 01:11:44.871397 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:44.870910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:58177)
0929 01:11:44.871086 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:44.871199 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 01:11:44.871685 22715 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 01:11:44.874859 22607 raft_consensus.cc:348] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:44.876273 22607 raft_consensus.cc:729] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: abff705fff9346c9a09fb1dc9c46af69, State: Initialized, Role: FOLLOWER
I20260929 01:11:44.876976 22607 consensus_queue.cc:260] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:44.878515 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:44.878189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:60279)
0929 01:11:44.878331 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:44.878409 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 01:11:44.878683 22607 ts_tablet_manager.cc:1428] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69: Time spent starting tablet: real 0.015s user 0.009s sys 0.008s
I20260929 01:11:44.879719 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:44.879303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:53899)
0929 01:11:44.879469 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:44.879562 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 01:11:45.192538 22718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:11:45.192829 22718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:11:45.193004 22718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:11:45.200416 22718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:11:45.200562 22718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:11:45.200755 22718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:11:45.210608 22718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:40439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=41895
--tserver_master_addrs=127.5.198.254:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:11:45.211930 22718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:11:45.213258 22718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:11:45.226883 22724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:45.226955 22727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:45.226915 22725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:11:45.227725 22718 server_base.cc:1034] running on GCE node
I20260929 01:11:45.228299 22718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:11:45.229372 22718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:11:45.230569 22718 hybrid_clock.cc:648] HybridClock initialized: now 1790644305230522 us; error 36 us; skew 500 ppm
I20260929 01:11:45.230904 22718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:11:45 dist-test-slave-41cl krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644305, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:11:45.234203 22718 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:11:45.235702 22718 webserver.cc:460] Webserver started at http://127.5.198.194:41895/ using document root <none> and password file <none>
I20260929 01:11:45.236644 22718 fs_manager.cc:359] Metadata directory not provided
I20260929 01:11:45.236806 22718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:11:45.242411 22718 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:11:45.245404 22734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:45.246811 22718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:11:45.247042 22718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "ef8ca4430404475a90195e362b3bae28"
format_stamp: "Formatted at 2026-09-29 01:10:59 on dist-test-slave-41cl"
server_key: "98f98ec1fb99500ef655dccfa8b4c636"
server_key_iv: "10d2d537ea6fae3f7e2623cfb1162b06"
server_key_version: "encryptionkey@0"
I20260929 01:11:45.248083 22718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:11:45.271495 22718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:11:45.276120 22718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:11:45.276408 22718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:11:45.277499 22718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:11:45.278872 22718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:11:45.279006 22718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:45.279125 22718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:11:45.279207 22718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:45.313491 22718 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:40439
I20260929 01:11:45.313575 22846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:40439 every 8 connection(s)
I20260929 01:11:45.314939 22718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:11:45.319795 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:45.317960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:37149)
0929 01:11:45.318681 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:45.318929 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":276,"threads_started":1}
W20260929 01:11:45.320600 22847 heartbeater.cc:644] Failed to heartbeat to 127.5.198.254:38197 (0 consecutive failures): Network error: Failed to ping master at 127.5.198.254:38197: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
I20260929 01:11:45.320832 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 22718
I20260929 01:11:45.321550 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:45.321239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:36345)
0929 01:11:45.321383 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:45.321449 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 01:11:45.321583 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:45563
--local_ip_for_outbound_sockets=127.5.198.195
--tserver_master_addrs=127.5.198.254:38197
--webserver_port=36103
--webserver_interface=127.5.198.195
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:11:45.322535 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:45.322229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:50501)
0929 01:11:45.322351 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:45.322431 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 01:11:45.322813 22847 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:11:45.654649 22850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:11:45.654925 22850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:11:45.655020 22850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:11:45.662212 22850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:11:45.662314 22850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:11:45.662431 22850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:11:45.672091 22850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:45563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=36103
--tserver_master_addrs=127.5.198.254:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:11:45.673323 22850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:11:45.674726 22850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:11:45.688867 22855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:45.688908 22858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:11:45.688936 22856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:11:45.689893 22850 server_base.cc:1034] running on GCE node
I20260929 01:11:45.690485 22850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:11:45.691421 22850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:11:45.692664 22850 hybrid_clock.cc:648] HybridClock initialized: now 1790644305692612 us; error 60 us; skew 500 ppm
I20260929 01:11:45.693028 22850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:11:45 dist-test-slave-41cl krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644305, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:11:45.696264 22850 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:11:45.697824 22850 webserver.cc:460] Webserver started at http://127.5.198.195:36103/ using document root <none> and password file <none>
I20260929 01:11:45.698519 22850 fs_manager.cc:359] Metadata directory not provided
I20260929 01:11:45.698662 22850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:11:45.704013 22850 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:11:45.706930 22865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:45.708293 22850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 01:11:45.708521 22850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "80f5a55a3a2442dab20236503cef7729"
format_stamp: "Formatted at 2026-09-29 01:11:00 on dist-test-slave-41cl"
server_key: "064fd0f3193fe7d19769e3294bdf7ebd"
server_key_iv: "d5475b670a9da224a8df2fa0d0c38b36"
server_key_version: "encryptionkey@0"
I20260929 01:11:45.709517 22850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:11:45.735168 22850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:11:45.739521 22850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:11:45.739805 22850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:11:45.740803 22850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:11:45.742177 22850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:11:45.742295 22850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:45.742417 22850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:11:45.742498 22850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:11:45.776377 22850 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:45563
I20260929 01:11:45.776440 22977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:45563 every 8 connection(s)
I20260929 01:11:45.777837 22850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:11:45.782820 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:45.780935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:53617)
0929 01:11:45.781750 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:45.781989 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":354,"threads_started":1}
W20260929 01:11:45.783464 22978 heartbeater.cc:644] Failed to heartbeat to 127.5.198.254:38197 (0 consecutive failures): Network error: Failed to ping master at 127.5.198.254:38197: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
I20260929 01:11:45.784253 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:45.783916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:37277)
0929 01:11:45.784069 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:45.784133 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 01:11:45.785071 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:45.784764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:38081)
0929 01:11:45.784872 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:45.784939 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 01:11:45.785337 22978 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 01:11:45.785568 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 22850
I20260929 01:11:45.881690 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:45.880822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:53301)
0929 01:11:45.881327 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:45.881467 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":259,"threads_started":1}
I20260929 01:11:46.324131 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:46.323317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:55097)
0929 01:11:46.323803 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:46.323890 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":258,"threads_started":1}
I20260929 01:11:46.434037 22984 raft_consensus.cc:482] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:11:46.434290 22984 raft_consensus.cc:504] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:46.435985 22984 leader_election.cc:288] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80f5a55a3a2442dab20236503cef7729 (127.5.198.195:45563), ef8ca4430404475a90195e362b3bae28 (127.5.198.194:40439)
Sep 29 01:11:46 dist-test-slave-41cl krb5kdc[21395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644304, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
Sep 29 01:11:46 dist-test-slave-41cl krb5kdc[21395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644304, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:11:46.459378 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:46.436682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:33003 (local address 127.5.198.194:40439)
0929 01:11:46.438411 (+ 1729us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:46.438426 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:46.438467 (+ 41us) server_negotiation.cc:407] Connection header received
0929 01:11:46.440645 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:46.440656 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:46.441105 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:46.441428 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:11:46.443535 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:46.445527 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:46.447498 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:46.448118 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:46.448311 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:46.452232 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:11:46.452240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:11:46.452255 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:11:46.452289 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:11:46.455389 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:46.456221 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:46.456230 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:46.456241 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:46.456437 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:46.457330 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:46.457336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:46.457340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:46.457906 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:11:46.458086 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:46.458633 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 01:11:46.458961 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1551,"thread_start_us":1385,"threads_started":1}
I20260929 01:11:46.459446 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:46.437096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:51411 (local address 127.5.198.195:45563)
0929 01:11:46.437681 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0929 01:11:46.437700 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:11:46.437746 (+ 46us) server_negotiation.cc:407] Connection header received
0929 01:11:46.440076 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:11:46.440089 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:11:46.440634 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:11:46.440983 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:11:46.443535 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:46.445527 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:46.447479 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:11:46.448104 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:11:46.448318 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:46.452507 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:11:46.452517 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:11:46.452536 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:11:46.452576 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:11:46.455568 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:46.456370 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:46.456378 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:46.456390 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:46.456600 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:11:46.457033 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:11:46.457037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:11:46.457040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:11:46.457606 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:11:46.457753 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 01:11:46.458627 (+ 874us) server_negotiation.cc:299] Negotiation successful
0929 01:11:46.459004 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":261,"threads_started":1}
I20260929 01:11:46.459563 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:46.436329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:45563 (local address 127.5.198.193:51411)
0929 01:11:46.436984 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:46.437022 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:11:46.437672 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:11:46.441151 (+ 3479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:11:46.441159 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:11:46.442150 (+ 991us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:11:46.443322 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:46.443350 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:46.445789 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:46.445796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:46.447337 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:46.447351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:46.448329 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:46.448336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:46.448699 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:46.449505 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:11:46.449528 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:11:46.452213 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:11:46.455709 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:11:46.455717 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:11:46.455721 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:11:46.456225 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:11:46.456731 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:11:46.456734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:11:46.456737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:11:46.456903 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:11:46.457766 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:11:46.457773 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:11:46.458146 (+ 373us) client_negotiation.cc:769] Sending connection context
0929 01:11:46.458418 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 01:11:46.459137 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":517,"spinlock_wait_cycles":31104,"thread_start_us":793,"threads_started":1}
I20260929 01:11:46.460688 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:46.436349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:40439 (local address 127.5.198.193:33003)
0929 01:11:46.437423 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:46.437444 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:11:46.437671 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:11:46.441401 (+ 3730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:11:46.441406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:11:46.442151 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:11:46.443322 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:46.443367 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:46.445759 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:46.445765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:46.447344 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:11:46.447357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:11:46.448356 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:11:46.448362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:11:46.448697 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:11:46.449559 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:11:46.449577 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:11:46.451935 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:11:46.455544 (+ 3609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:11:46.455552 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:11:46.455566 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:11:46.456082 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:11:46.456603 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:11:46.456607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:11:46.456610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:11:46.456857 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:11:46.458041 (+ 1184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:11:46.458044 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:11:46.458171 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 01:11:46.458420 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 01:11:46.458987 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"mutex_wait_us":269,"spinlock_wait_cycles":21632,"thread_start_us":297,"threads_started":1}
W20260929 01:11:46.462744 22604 leader_election.cc:341] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80f5a55a3a2442dab20236503cef7729 (127.5.198.195:45563): Not found: Tablet not found: c77ad59ebc6345148c6327725c0aacd5
W20260929 01:11:46.463171 22601 leader_election.cc:341] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef8ca4430404475a90195e362b3bae28 (127.5.198.194:40439): Not found: Tablet not found: c77ad59ebc6345148c6327725c0aacd5
I20260929 01:11:46.463286 22601 leader_election.cc:302] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abff705fff9346c9a09fb1dc9c46af69; no voters: 80f5a55a3a2442dab20236503cef7729, ef8ca4430404475a90195e362b3bae28
I20260929 01:11:46.463635 22984 raft_consensus.cc:2726] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:11:46.786746 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:46.785807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:57869)
0929 01:11:46.786353 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:46.786493 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":287,"threads_started":1}
I20260929 01:11:46.883004 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:46.882554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:33343)
0929 01:11:46.882737 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:46.882848 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 01:11:47.325809 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:47.325095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:47281)
0929 01:11:47.325568 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:47.325654 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":229,"threads_started":1}
I20260929 01:11:47.788491 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:47.787724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:42641)
0929 01:11:47.788219 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:47.788307 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":287,"threads_started":1}
I20260929 01:11:47.884896 22992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:47.883962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:43771)
0929 01:11:47.884605 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:47.884717 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":248,"threads_started":1}
I20260929 01:11:48.327597 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:48.326798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:45359)
0929 01:11:48.327332 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:48.327421 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":324,"threads_started":1}
I20260929 01:11:48.350028 22994 raft_consensus.cc:482] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:11:48.350226 22994 raft_consensus.cc:504] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:48.351007 22994 leader_election.cc:288] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80f5a55a3a2442dab20236503cef7729 (127.5.198.195:45563), ef8ca4430404475a90195e362b3bae28 (127.5.198.194:40439)
W20260929 01:11:48.352231 22604 leader_election.cc:341] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80f5a55a3a2442dab20236503cef7729 (127.5.198.195:45563): Not found: Tablet not found: c77ad59ebc6345148c6327725c0aacd5
W20260929 01:11:48.352514 22601 leader_election.cc:341] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef8ca4430404475a90195e362b3bae28 (127.5.198.194:40439): Not found: Tablet not found: c77ad59ebc6345148c6327725c0aacd5
I20260929 01:11:48.352627 22601 leader_election.cc:302] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abff705fff9346c9a09fb1dc9c46af69; no voters: 80f5a55a3a2442dab20236503cef7729, ef8ca4430404475a90195e362b3bae28
I20260929 01:11:48.352887 22994 raft_consensus.cc:2726] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:11:48.790216 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:48.789427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:55433)
0929 01:11:48.789956 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:48.790048 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":256,"threads_started":1}
I20260929 01:11:48.886682 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:48.885919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:33539)
0929 01:11:48.886409 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:48.886513 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":253,"threads_started":1}
I20260929 01:11:49.329669 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:49.328774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:57223)
0929 01:11:49.329350 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:49.329464 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":377,"threads_started":1}
I20260929 01:11:49.791931 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:49.791202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:51667)
0929 01:11:49.791675 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:49.791761 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":246,"threads_started":1}
I20260929 01:11:49.888967 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:49.887933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:47169)
0929 01:11:49.888440 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:49.888605 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":267,"threads_started":1}
I20260929 01:11:50.135483 23000 raft_consensus.cc:482] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:11:50.135676 23000 raft_consensus.cc:504] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:50.136368 23000 leader_election.cc:288] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80f5a55a3a2442dab20236503cef7729 (127.5.198.195:45563), ef8ca4430404475a90195e362b3bae28 (127.5.198.194:40439)
W20260929 01:11:50.137401 22604 leader_election.cc:341] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80f5a55a3a2442dab20236503cef7729 (127.5.198.195:45563): Not found: Tablet not found: c77ad59ebc6345148c6327725c0aacd5
W20260929 01:11:50.137571 22601 leader_election.cc:341] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef8ca4430404475a90195e362b3bae28 (127.5.198.194:40439): Not found: Tablet not found: c77ad59ebc6345148c6327725c0aacd5
I20260929 01:11:50.137670 22601 leader_election.cc:302] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abff705fff9346c9a09fb1dc9c46af69; no voters: 80f5a55a3a2442dab20236503cef7729, ef8ca4430404475a90195e362b3bae28
I20260929 01:11:50.137893 23000 raft_consensus.cc:2726] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:11:50.331528 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:50.330754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:46235)
0929 01:11:50.331245 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:50.331336 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":282,"threads_started":1}
I20260929 01:11:50.793787 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:50.793067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:42399)
0929 01:11:50.793540 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:50.793621 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":266,"threads_started":1}
I20260929 01:11:50.890728 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:50.889900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:40633)
0929 01:11:50.890436 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:50.890555 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":271,"threads_started":1}
I20260929 01:11:51.333375 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:51.332450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:33615)
0929 01:11:51.333107 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:51.333200 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":270,"threads_started":1}
I20260929 01:11:51.795593 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:51.794812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:58093)
0929 01:11:51.795334 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:51.795415 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":310,"threads_started":1}
I20260929 01:11:51.892680 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:51.891829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:37549)
0929 01:11:51.892290 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:51.892391 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":255,"threads_started":1}
I20260929 01:11:52.225271 23007 raft_consensus.cc:482] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:11:52.225555 23007 raft_consensus.cc:504] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:52.226370 23007 leader_election.cc:288] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80f5a55a3a2442dab20236503cef7729 (127.5.198.195:45563), ef8ca4430404475a90195e362b3bae28 (127.5.198.194:40439)
W20260929 01:11:52.227396 22601 leader_election.cc:341] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef8ca4430404475a90195e362b3bae28 (127.5.198.194:40439): Not found: Tablet not found: c77ad59ebc6345148c6327725c0aacd5
W20260929 01:11:52.227607 22604 leader_election.cc:341] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80f5a55a3a2442dab20236503cef7729 (127.5.198.195:45563): Not found: Tablet not found: c77ad59ebc6345148c6327725c0aacd5
I20260929 01:11:52.227742 22604 leader_election.cc:302] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abff705fff9346c9a09fb1dc9c46af69; no voters: 80f5a55a3a2442dab20236503cef7729, ef8ca4430404475a90195e362b3bae28
I20260929 01:11:52.228065 23007 raft_consensus.cc:2726] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:11:52.335328 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:52.334460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:48315)
0929 01:11:52.335040 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:52.335141 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":248,"threads_started":1}
I20260929 01:11:52.797298 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:52.796527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:47105)
0929 01:11:52.797020 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:52.797104 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":236,"threads_started":1}
I20260929 01:11:52.894490 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:52.893668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:47623)
0929 01:11:52.894225 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:52.894325 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":299,"threads_started":1}
I20260929 01:11:53.337231 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:53.336380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:36075)
0929 01:11:53.336936 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:53.337039 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":329,"threads_started":1}
I20260929 01:11:53.798835 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:53.798156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:44317)
0929 01:11:53.798574 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:53.798658 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":221,"threads_started":1}
I20260929 01:11:53.896344 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:53.895507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:48071)
0929 01:11:53.896070 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:53.896162 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":297,"threads_started":1}
I20260929 01:11:54.338914 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:54.338186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:42539)
0929 01:11:54.338668 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:54.338751 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":227,"threads_started":1}
I20260929 01:11:54.800448 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:54.799688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:51919)
0929 01:11:54.800214 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:54.800302 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":238,"threads_started":1}
I20260929 01:11:54.898149 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:54.897426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:46001)
0929 01:11:54.897856 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:54.897943 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":224,"threads_started":1}
I20260929 01:11:55.340705 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:55.339904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:38173)
0929 01:11:55.340364 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:55.340440 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":260,"threads_started":1}
I20260929 01:11:55.717483 23018 raft_consensus.cc:482] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:11:55.717672 23018 raft_consensus.cc:504] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abff705fff9346c9a09fb1dc9c46af69" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37583 } } peers { permanent_uuid: "80f5a55a3a2442dab20236503cef7729" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 45563 } } peers { permanent_uuid: "ef8ca4430404475a90195e362b3bae28" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40439 } }
I20260929 01:11:55.718295 23018 leader_election.cc:288] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80f5a55a3a2442dab20236503cef7729 (127.5.198.195:45563), ef8ca4430404475a90195e362b3bae28 (127.5.198.194:40439)
W20260929 01:11:55.719148 22604 leader_election.cc:341] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80f5a55a3a2442dab20236503cef7729 (127.5.198.195:45563): Not found: Tablet not found: c77ad59ebc6345148c6327725c0aacd5
W20260929 01:11:55.719344 22601 leader_election.cc:341] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef8ca4430404475a90195e362b3bae28 (127.5.198.194:40439): Not found: Tablet not found: c77ad59ebc6345148c6327725c0aacd5
I20260929 01:11:55.719477 22601 leader_election.cc:302] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abff705fff9346c9a09fb1dc9c46af69; no voters: 80f5a55a3a2442dab20236503cef7729, ef8ca4430404475a90195e362b3bae28
I20260929 01:11:55.719699 23018 raft_consensus.cc:2726] T c77ad59ebc6345148c6327725c0aacd5 P abff705fff9346c9a09fb1dc9c46af69 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:11:55.802245 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:55.801416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:34035)
0929 01:11:55.801905 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:55.801999 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":262,"threads_started":1}
I20260929 01:11:55.900070 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:55.899181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:34215)
0929 01:11:55.899690 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:55.899794 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":272,"threads_started":1}
I20260929 01:11:56.342414 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:56.341646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:52539)
0929 01:11:56.342164 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:56.342243 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":223,"threads_started":1}
I20260929 01:11:56.803843 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:56.803129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:53841)
0929 01:11:56.803580 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:56.803688 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":215,"threads_started":1}
I20260929 01:11:56.901721 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:56.900999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:55751)
0929 01:11:56.901463 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:56.901562 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":260,"threads_started":1}
I20260929 01:11:57.344056 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:57.343377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:49705)
0929 01:11:57.343789 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:57.343866 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":222,"threads_started":1}
I20260929 01:11:57.805476 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:57.804782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:51531)
0929 01:11:57.805227 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:57.805305 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":252,"threads_started":1}
I20260929 01:11:57.903440 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:57.902717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:44665)
0929 01:11:57.903169 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:57.903248 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":233,"threads_started":1}
I20260929 01:11:58.345890 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:58.345069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:51537)
0929 01:11:58.345640 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:58.345729 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":289,"threads_started":1}
I20260929 01:11:58.807158 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:58.806400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:44271)
0929 01:11:58.806857 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:58.806949 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":237,"threads_started":1}
I20260929 01:11:58.905319 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:58.904451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:60841)
0929 01:11:58.905018 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:58.905129 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":253,"threads_started":1}
I20260929 01:11:59.347649 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:59.346816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:54925)
0929 01:11:59.347356 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:59.347451 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":273,"threads_started":1}
I20260929 01:11:59.808878 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:59.808115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:40811)
0929 01:11:59.808600 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:59.808690 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":218,"threads_started":1}
I20260929 01:11:59.907342 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:11:59.906570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.193:37235)
0929 01:11:59.907075 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 01:11:59.907164 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":236,"threads_started":1}
I20260929 01:12:00.349884 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:00.349119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.194:58257)
0929 01:12:00.349627 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:00.349722 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":260,"threads_started":1}
I20260929 01:12:00.792680 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 22545
I20260929 01:12:00.810765 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:00.810024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:38197 (local address 127.5.198.195:58157)
0929 01:12:00.810512 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:00.810616 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.254:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":248,"threads_started":1}
I20260929 01:12:00.814393 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:00.793675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48050 (local address 127.5.198.193:37583)
0929 01:12:00.794376 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:00.794388 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:00.794426 (+ 38us) server_negotiation.cc:407] Connection header received
0929 01:12:00.794524 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:00.794531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:00.794847 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:00.795014 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:00.796936 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:00.797971 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:00.799269 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:00.799869 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:00.800032 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:00.808432 (+ 8400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:00.808444 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:00.808523 (+ 79us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:00.808575 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:00.811717 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:00.812374 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:00.812380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:00.812382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:00.812603 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:00.812946 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:00.812951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:00.812956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:00.813349 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:00.813596 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:00.813833 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 01:12:00.814075 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":428,"threads_started":1}
I20260929 01:12:00.815618 22710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:00.905786 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 22545
I20260929 01:12:00.924427 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 22718
I20260929 01:12:00.938199 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:00.925283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.5.198.194:40439)
0929 01:12:00.925811 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:00.925816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:00.925831 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:12:00.925886 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:00.925890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:00.925947 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:00.926038 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:00.927657 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:00.928434 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:00.929696 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:00.930109 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:00.930197 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:00.932746 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:00.932751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:00.932754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:00.932786 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:00.935440 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:00.936062 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:00.936066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:00.936068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:00.936201 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:00.936766 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:00.936770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:00.936772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:00.937043 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:00.937279 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:00.937610 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 01:12:00.937776 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":294,"threads_started":1}
I20260929 01:12:00.938920 22842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:01.021770 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 22718
I20260929 01:12:01.038915 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 22850
I20260929 01:12:01.052169 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:01.039815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.5.198.195:45563)
0929 01:12:01.040419 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:01.040424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:01.040441 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:12:01.040556 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:01.040560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:01.040633 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:01.040724 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:01.042028 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:01.042802 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:01.043990 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:01.044384 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:01.044620 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:01.046998 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:01.047004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:01.047007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:01.047037 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:01.049786 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:01.050424 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:01.050429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:01.050431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:01.050571 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:01.050895 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:01.050898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:01.050901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:01.051150 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:01.051351 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:01.051580 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 01:12:01.051735 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":340,"threads_started":1}
I20260929 01:12:01.052979 22973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:01.135799 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 22850
2026-09-29T01:12:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64457 ms)
[----------] 30 tests from SecurityITest (297807 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:01 dist-test-slave-41cl krb5kdc[23043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:01 dist-test-slave-41cl krb5kdc[23043](info): set up 2 sockets
Sep 29 01:12:01 dist-test-slave-41cl krb5kdc[23043](info): commencing operation
krb5kdc: starting...
W20260929 01:12:02.252744 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:02 dist-test-slave-41cl krb5kdc[23043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:12:02.497270 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:39261
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:39261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:02.826112 23056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:02.826399 23056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:02.826566 23056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:02.833714 23056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:02.833809 23056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:02.833909 23056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:02.834015 23056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:02.834095 23056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:02.843802 23056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:39261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:39261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:02.845197 23056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:02.846477 23056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:02.858263 23064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:02.858335 23061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:02.858284 23062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:02.859575 23056 server_base.cc:1034] running on GCE node
I20260929 01:12:02.860430 23056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:02.861882 23056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:02.863159 23056 hybrid_clock.cc:648] HybridClock initialized: now 1790644322863067 us; error 71 us; skew 500 ppm
I20260929 01:12:02.863561 23056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:02 dist-test-slave-41cl krb5kdc[23043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644322, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:02.866902 23056 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:12:02.868539 23056 webserver.cc:460] Webserver started at http://127.5.198.254:40131/ using document root <none> and password file <none>
I20260929 01:12:02.869222 23056 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:02.869369 23056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:02.869688 23056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:02.872659 23056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "d951129c4cea4ffea6cd64fb591a8490"
format_stamp: "Formatted at 2026-09-29 01:12:02 on dist-test-slave-41cl"
server_key: "cda8f9f83c47592dd45cd84a9b3f79fe"
server_key_iv: "b6f705219b111fa6f425e8a07389ca40"
server_key_version: "encryptionkey@0"
I20260929 01:12:02.873468 23056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "d951129c4cea4ffea6cd64fb591a8490"
format_stamp: "Formatted at 2026-09-29 01:12:02 on dist-test-slave-41cl"
server_key: "cda8f9f83c47592dd45cd84a9b3f79fe"
server_key_iv: "b6f705219b111fa6f425e8a07389ca40"
server_key_version: "encryptionkey@0"
I20260929 01:12:02.878664 23056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:12:02.882249 23071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:02.883599 23056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:12:02.883790 23056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "d951129c4cea4ffea6cd64fb591a8490"
format_stamp: "Formatted at 2026-09-29 01:12:02 on dist-test-slave-41cl"
server_key: "cda8f9f83c47592dd45cd84a9b3f79fe"
server_key_iv: "b6f705219b111fa6f425e8a07389ca40"
server_key_version: "encryptionkey@0"
I20260929 01:12:02.883987 23056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:02.899665 23056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:02.903695 23056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:02.903980 23056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:02.923190 23056 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:39261
I20260929 01:12:02.923271 23122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:39261 every 8 connection(s)
I20260929 01:12:02.924573 23056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:02.928141 23123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:02.931346 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 23056
I20260929 01:12:02.931559 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:02.932436 5915 external_mini_cluster.cc:1428] Setting key e782d3d2166d7307fe76f260b11553d4
I20260929 01:12:02.940856 23123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490: Bootstrap starting.
I20260929 01:12:02.944530 23123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:02.945739 23123 log.cc:824] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:02.948763 23123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490: No bootstrap required, opened a new log
Sep 29 01:12:02 dist-test-slave-41cl krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:12:02.958218 23123 raft_consensus.cc:348] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d951129c4cea4ffea6cd64fb591a8490" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39261 } }
I20260929 01:12:02.958762 23123 raft_consensus.cc:374] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:02.958905 23123 raft_consensus.cc:729] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d951129c4cea4ffea6cd64fb591a8490, State: Initialized, Role: FOLLOWER
I20260929 01:12:02.959530 23123 consensus_queue.cc:260] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d951129c4cea4ffea6cd64fb591a8490" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39261 } }
I20260929 01:12:02.959777 23123 raft_consensus.cc:388] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:02.959817 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:02.934674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.5.198.254:39261)
0929 01:12:02.935574 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:02.935589 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:02.936020 (+ 431us) server_negotiation.cc:407] Connection header received
0929 01:12:02.939090 (+ 3070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:02.939129 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:02.939634 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:02.940058 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:02.941582 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:02.943630 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:02.945091 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:02.946140 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:02.946365 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:02.951117 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:02.951127 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:02.951141 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:02.951186 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:02.954954 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:02.955890 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:02.955941 (+ 51us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:02.955947 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:02.956144 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:02.956615 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:02.956619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:02.956621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:02.957201 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:02.957417 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:02.957917 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 01:12:02.958455 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":415,"threads_started":1}
I20260929 01:12:02.959913 23123 raft_consensus.cc:482] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:02.960091 23123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:02.962534 23123 raft_consensus.cc:504] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d951129c4cea4ffea6cd64fb591a8490" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39261 } }
I20260929 01:12:02.963061 23123 leader_election.cc:302] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d951129c4cea4ffea6cd64fb591a8490; no voters:
I20260929 01:12:02.963737 23123 leader_election.cc:288] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:02.963814 23128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:02.964576 23128 raft_consensus.cc:686] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [term 1 LEADER]: Becoming Leader. State: Replica: d951129c4cea4ffea6cd64fb591a8490, State: Running, Role: LEADER
I20260929 01:12:02.965196 23128 consensus_queue.cc:237] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d951129c4cea4ffea6cd64fb591a8490" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39261 } }
I20260929 01:12:02.965848 23123 sys_catalog.cc:564] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:02.968091 23130 sys_catalog.cc:455] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d951129c4cea4ffea6cd64fb591a8490. Latest consensus state: current_term: 1 leader_uuid: "d951129c4cea4ffea6cd64fb591a8490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d951129c4cea4ffea6cd64fb591a8490" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39261 } } }
I20260929 01:12:02.968538 23130 sys_catalog.cc:458] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:02.968639 23129 sys_catalog.cc:455] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d951129c4cea4ffea6cd64fb591a8490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d951129c4cea4ffea6cd64fb591a8490" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 39261 } } }
I20260929 01:12:02.968931 23129 sys_catalog.cc:458] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:02.973775 23134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:02.979581 23134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:02.988247 23134 catalog_manager.cc:1347] Generated new cluster ID: 65e1cbc1ad7e45379e5f9e9819641ca9
I20260929 01:12:02.988361 23134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:03.006283 23134 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:03.008204 23134 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:03.015561 23134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490: Generated new TSK 0
I20260929 01:12:03.016626 23134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:12:03.085135 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:39261
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:03.412957 23149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:03.413223 23149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:03.413381 23149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:03.420703 23149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:03.420804 23149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:03.420979 23149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:12:03.430827 23149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:39261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:03.432123 23149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:03.433446 23149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:03.446751 23155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:03.446700 23157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:03.446789 23154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:03.448184 23149 server_base.cc:1034] running on GCE node
I20260929 01:12:03.448880 23149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:03.449867 23149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:03.451131 23149 hybrid_clock.cc:648] HybridClock initialized: now 1790644323451061 us; error 57 us; skew 500 ppm
I20260929 01:12:03.451501 23149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:03 dist-test-slave-41cl krb5kdc[23043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644323, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:03.454927 23149 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:12:03.456612 23149 webserver.cc:460] Webserver started at http://127.5.198.193:40971/ using document root <none> and password file <none>
I20260929 01:12:03.457333 23149 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:03.457481 23149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:03.457804 23149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:03.460851 23149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "e16f7e9b588440409a6c662aa50d9436"
format_stamp: "Formatted at 2026-09-29 01:12:03 on dist-test-slave-41cl"
server_key: "8fd8cfb227f3d7c71f7bdd2bc5422d91"
server_key_iv: "5695b1938fcefe6aa395cfc96adb44bb"
server_key_version: "encryptionkey@0"
I20260929 01:12:03.461642 23149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "e16f7e9b588440409a6c662aa50d9436"
format_stamp: "Formatted at 2026-09-29 01:12:03 on dist-test-slave-41cl"
server_key: "8fd8cfb227f3d7c71f7bdd2bc5422d91"
server_key_iv: "5695b1938fcefe6aa395cfc96adb44bb"
server_key_version: "encryptionkey@0"
I20260929 01:12:03.467154 23149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 01:12:03.470700 23164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:03.472263 23149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:12:03.472405 23149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "e16f7e9b588440409a6c662aa50d9436"
format_stamp: "Formatted at 2026-09-29 01:12:03 on dist-test-slave-41cl"
server_key: "8fd8cfb227f3d7c71f7bdd2bc5422d91"
server_key_iv: "5695b1938fcefe6aa395cfc96adb44bb"
server_key_version: "encryptionkey@0"
I20260929 01:12:03.472661 23149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:03.490218 23149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:03.494449 23149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:03.494688 23149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:03.495615 23149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:03.497014 23149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:03.497095 23149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:03.497181 23149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:03.497224 23149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:03.529831 23149 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:37461
I20260929 01:12:03.529875 23276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:37461 every 8 connection(s)
I20260929 01:12:03.531111 23149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:12:03.539285 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 23149
I20260929 01:12:03.539484 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:12:03.540226 5915 external_mini_cluster.cc:1428] Setting key a5f2e5980dd9fded3551f701ef6807bb
Sep 29 01:12:03 dist-test-slave-41cl krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644323, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:12:03.552848 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:03.533645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:41045 (local address 127.5.198.254:39261)
0929 01:12:03.534136 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:03.534143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:03.534988 (+ 845us) server_negotiation.cc:407] Connection header received
0929 01:12:03.536973 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:03.536978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:03.537074 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:03.537193 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:03.538701 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:03.539812 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:03.541779 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:03.542198 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:03.542299 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:03.545982 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:03.545991 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:03.545998 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:03.546041 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:03.549793 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:03.550511 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:03.550520 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:03.550526 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:03.550675 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:03.551133 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:03.551137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:03.551139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:03.551395 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:03.551554 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:03.552233 (+ 679us) server_negotiation.cc:299] Negotiation successful
0929 01:12:03.552461 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":250,"threads_started":1}
I20260929 01:12:03.553817 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:03.534097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39261 (local address 127.5.198.193:41045)
0929 01:12:03.534803 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:03.534847 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:03.536701 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:03.537432 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:03.537440 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:03.537967 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:03.538521 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:03.538538 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:03.539982 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:03.539987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:03.541617 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:03.541632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:03.542427 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:03.542433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:03.542637 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:03.543332 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:12:03.543354 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:03.545751 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:03.549936 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:03.549943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:03.549956 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:03.550366 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:03.550786 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:03.550789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:03.550791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:03.550960 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:03.551526 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:03.551534 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:03.551873 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 01:12:03.552131 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 01:12:03.552696 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":291,"threads_started":1}
I20260929 01:12:03.555361 23277 heartbeater.cc:347] Connected to a master server at 127.5.198.254:39261
I20260929 01:12:03.555676 23277 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:03.556541 23277 heartbeater.cc:511] Master 127.5.198.254:39261 requested a full tablet report, sending...
I20260929 01:12:03.558926 23088 ts_manager.cc:194] Registered new tserver with Master: e16f7e9b588440409a6c662aa50d9436 (127.5.198.193:37461)
I20260929 01:12:03.561055 23088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:41045
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:12:03.599743 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:39261
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:03.922876 23283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:03.923144 23283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:03.923303 23283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:03.930614 23283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:03.930711 23283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:03.930877 23283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:12:03.940307 23283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:39261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:03.941519 23283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:03.942775 23283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:03.955209 23288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:03.955271 23289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:03.955305 23291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:03.956522 23283 server_base.cc:1034] running on GCE node
I20260929 01:12:03.957216 23283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:03.958271 23283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:03.959522 23283 hybrid_clock.cc:648] HybridClock initialized: now 1790644323959462 us; error 65 us; skew 500 ppm
I20260929 01:12:03.959860 23283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:03 dist-test-slave-41cl krb5kdc[23043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644323, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:03.963202 23283 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:12:03.964812 23283 webserver.cc:460] Webserver started at http://127.5.198.194:38581/ using document root <none> and password file <none>
I20260929 01:12:03.965488 23283 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:03.965642 23283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:03.965960 23283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:03.968919 23283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "81f14c884c9e4c16942f7323c020892a"
format_stamp: "Formatted at 2026-09-29 01:12:03 on dist-test-slave-41cl"
server_key: "42e419a5d6722aafc7bb52e11dd5325c"
server_key_iv: "439f4654fd3afd862a4806fd967f5843"
server_key_version: "encryptionkey@0"
I20260929 01:12:03.969709 23283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "81f14c884c9e4c16942f7323c020892a"
format_stamp: "Formatted at 2026-09-29 01:12:03 on dist-test-slave-41cl"
server_key: "42e419a5d6722aafc7bb52e11dd5325c"
server_key_iv: "439f4654fd3afd862a4806fd967f5843"
server_key_version: "encryptionkey@0"
I20260929 01:12:03.975018 23283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:12:03.978523 23298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:03.979848 23283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:12:03.980042 23283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "81f14c884c9e4c16942f7323c020892a"
format_stamp: "Formatted at 2026-09-29 01:12:03 on dist-test-slave-41cl"
server_key: "42e419a5d6722aafc7bb52e11dd5325c"
server_key_iv: "439f4654fd3afd862a4806fd967f5843"
server_key_version: "encryptionkey@0"
I20260929 01:12:03.980240 23283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:03.992961 23283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:03.996919 23283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:03.997203 23283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:03.998113 23283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:03.999554 23283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:03.999655 23283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:03.999794 23283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:03.999863 23283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:04.033591 23283 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:38947
I20260929 01:12:04.033632 23410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:38947 every 8 connection(s)
I20260929 01:12:04.034861 23283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:12:04.043110 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 23283
I20260929 01:12:04.043285 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:12:04.043776 5915 external_mini_cluster.cc:1428] Setting key 68ce338ffc580085ed9178cb37ff1876
Sep 29 01:12:04 dist-test-slave-41cl krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644323, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:12:04.056887 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.037476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:60125 (local address 127.5.198.254:39261)
0929 01:12:04.037686 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:04.037690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:04.038830 (+ 1140us) server_negotiation.cc:407] Connection header received
0929 01:12:04.040751 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:04.040757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:04.040851 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:04.040962 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:04.042420 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.043425 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.045410 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.046092 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.046224 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.049627 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:04.049636 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:04.049644 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:04.049686 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:04.053325 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:04.054039 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:04.054048 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:04.054054 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:04.054219 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:04.054738 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:04.054746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:04.054751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:04.055237 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:04.055416 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:04.056125 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 01:12:04.056387 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 01:12:04.057809 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.037913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39261 (local address 127.5.198.194:60125)
0929 01:12:04.038660 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:04.038705 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:04.040422 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:04.041201 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:04.041209 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:04.041692 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:04.042246 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:04.042264 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.043591 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.043595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:04.045235 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:04.045249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.046301 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.046306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:04.046498 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.047189 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:12:04.047212 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:04.049380 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:04.053467 (+ 4087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:04.053473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:04.053485 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:04.053898 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:04.054345 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:04.054351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:04.054355 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:04.054545 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:04.055402 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:04.055408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:04.055744 (+ 336us) client_negotiation.cc:769] Sending connection context
0929 01:12:04.056015 (+ 271us) client_negotiation.cc:240] Negotiation successful
0929 01:12:04.056418 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":313,"threads_started":1}
I20260929 01:12:04.059406 23411 heartbeater.cc:347] Connected to a master server at 127.5.198.254:39261
I20260929 01:12:04.059746 23411 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:04.060633 23411 heartbeater.cc:511] Master 127.5.198.254:39261 requested a full tablet report, sending...
I20260929 01:12:04.062184 23088 ts_manager.cc:194] Registered new tserver with Master: 81f14c884c9e4c16942f7323c020892a (127.5.198.194:38947)
I20260929 01:12:04.062999 23088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:60125
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:12:04.100620 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:39261
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:04.428908 23416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:04.429185 23416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:04.429281 23416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:04.436443 23416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:04.436560 23416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:04.436683 23416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:12:04.446326 23416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:39261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:04.447507 23416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:04.448784 23416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:04.461684 23421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:04.461647 23422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:04.461925 23424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:04.463073 23416 server_base.cc:1034] running on GCE node
I20260929 01:12:04.463662 23416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:04.464659 23416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:04.465886 23416 hybrid_clock.cc:648] HybridClock initialized: now 1790644324465843 us; error 36 us; skew 500 ppm
I20260929 01:12:04.466372 23416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:04 dist-test-slave-41cl krb5kdc[23043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644324, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:04.469692 23416 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:12:04.471253 23416 webserver.cc:460] Webserver started at http://127.5.198.195:41811/ using document root <none> and password file <none>
I20260929 01:12:04.472030 23416 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:04.472138 23416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:04.472391 23416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:04.475226 23416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb"
format_stamp: "Formatted at 2026-09-29 01:12:04 on dist-test-slave-41cl"
server_key: "eb98ed20468afed325683e20639c9f42"
server_key_iv: "3f240f5dadc759c39322349410ae7101"
server_key_version: "encryptionkey@0"
I20260929 01:12:04.476083 23416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb"
format_stamp: "Formatted at 2026-09-29 01:12:04 on dist-test-slave-41cl"
server_key: "eb98ed20468afed325683e20639c9f42"
server_key_iv: "3f240f5dadc759c39322349410ae7101"
server_key_version: "encryptionkey@0"
I20260929 01:12:04.481509 23416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 01:12:04.485067 23431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:04.486397 23416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:12:04.486603 23416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb"
format_stamp: "Formatted at 2026-09-29 01:12:04 on dist-test-slave-41cl"
server_key: "eb98ed20468afed325683e20639c9f42"
server_key_iv: "3f240f5dadc759c39322349410ae7101"
server_key_version: "encryptionkey@0"
I20260929 01:12:04.486801 23416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:04.506354 23416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:04.510834 23416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:04.511185 23416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:04.512188 23416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:04.513600 23416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:04.513706 23416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:04.513861 23416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:04.513935 23416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:04.549006 23416 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:46189
I20260929 01:12:04.549098 23543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:46189 every 8 connection(s)
I20260929 01:12:04.550334 23416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:12:04.555183 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 23416
I20260929 01:12:04.555369 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:12:04.555858 5915 external_mini_cluster.cc:1428] Setting key c1b2c70a6ca0d4f90f42140a49b6b568
Sep 29 01:12:04 dist-test-slave-41cl krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644324, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:12:04.564720 23277 heartbeater.cc:503] Master 127.5.198.254:39261 was elected leader, sending a full tablet report...
I20260929 01:12:04.571295 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.552997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:40947 (local address 127.5.198.254:39261)
0929 01:12:04.553207 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:04.553213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:04.554398 (+ 1185us) server_negotiation.cc:407] Connection header received
0929 01:12:04.556378 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:04.556382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:04.556454 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:04.556600 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:04.558590 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.559571 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.561120 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.561518 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.561623 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.565136 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:04.565141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:04.565145 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:04.565202 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:04.568045 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:04.568788 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:04.568792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:04.568794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:04.568923 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:04.569353 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:04.569357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:04.569359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:04.569736 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:04.569900 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:04.570680 (+ 780us) server_negotiation.cc:299] Negotiation successful
0929 01:12:04.570920 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 01:12:04.572180 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.553531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:39261 (local address 127.5.198.195:40947)
0929 01:12:04.554229 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:04.554267 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:04.556068 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:04.556879 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:04.556894 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:04.557565 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:04.558381 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:04.558405 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.559713 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.559717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:04.560983 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:04.560995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.561740 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.561746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:04.561938 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.562636 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:12:04.562664 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:04.564918 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:04.568166 (+ 3248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:04.568173 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:04.568184 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:04.568668 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:04.569023 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:04.569026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:04.569028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:04.569229 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:04.569870 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:04.569877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:04.570279 (+ 402us) client_negotiation.cc:769] Sending connection context
0929 01:12:04.570592 (+ 313us) client_negotiation.cc:240] Negotiation successful
0929 01:12:04.571101 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":271,"threads_started":1}
I20260929 01:12:04.573451 23544 heartbeater.cc:347] Connected to a master server at 127.5.198.254:39261
I20260929 01:12:04.573745 23544 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:04.574410 23544 heartbeater.cc:511] Master 127.5.198.254:39261 requested a full tablet report, sending...
I20260929 01:12:04.575785 23088 ts_manager.cc:194] Registered new tserver with Master: 20b9f1a79d1f40c193c5023aa9f8ccfb (127.5.198.195:46189)
I20260929 01:12:04.576697 23088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:40947
I20260929 01:12:04.583355 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:12:04.600853 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.587932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32944 (local address 127.5.198.254:39261)
0929 01:12:04.588197 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:04.588202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:04.588630 (+ 428us) server_negotiation.cc:407] Connection header received
0929 01:12:04.588764 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:04.588768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:04.588831 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:04.588916 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:04.590365 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.591201 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.592754 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.593172 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.593280 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.595845 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:04.595850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:04.595853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:04.595889 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:04.598475 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:04.599139 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:04.599143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:04.599147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:04.599283 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:04.599663 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:04.599666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:04.599668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:04.599957 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:04.600161 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:04.600420 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 01:12:04.600611 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260929 01:12:04.605113 23088 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:12:04.612769 23088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:12:04.650543 23561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.642055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.5.198.193:37461)
0929 01:12:04.642874 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:04.642883 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:04.643685 (+ 802us) server_negotiation.cc:407] Connection header received
0929 01:12:04.643844 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:04.643851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:04.644042 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:04.644233 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:04.644877 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.645897 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.647926 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.649321 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.649509 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.649737 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:04.650017 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 01:12:04.650184 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":425,"threads_started":1}
I20260929 01:12:04.650622 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.641667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37461 (local address 127.0.0.1:54362)
0929 01:12:04.643546 (+ 1879us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:04.643560 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:04.643709 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:04.644249 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:04.644256 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:04.644293 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:04.644728 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:04.644736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.646020 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.646025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:04.647770 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:04.647782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.649568 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.649574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:04.649763 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.649810 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 01:12:04.649952 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 01:12:04.650161 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1773,"thread_start_us":1576,"threads_started":1}
I20260929 01:12:04.651276 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.626492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:38947 (local address 127.0.0.1:39584)
0929 01:12:04.638645 (+ 12153us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:04.638705 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:04.638964 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:04.642104 (+ 3140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:04.642112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:04.642154 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:04.644020 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:04.644029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.645942 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.645950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:04.648320 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:04.648334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.650369 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.650375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:04.650578 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.650627 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 01:12:04.650726 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 01:12:04.650842 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11775,"mutex_wait_us":176,"spinlock_wait_cycles":1408,"thread_start_us":14842,"threads_started":1}
I20260929 01:12:04.651432 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.626840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:46189 (local address 127.0.0.1:54640)
0929 01:12:04.641132 (+ 14292us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:04.641151 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:04.641272 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:04.642104 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:04.642112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:04.642160 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:04.644504 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:04.644522 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.646397 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.646402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:04.648566 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:04.648579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.650650 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.650658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:04.650832 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.650863 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 01:12:04.650942 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 01:12:04.651075 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14169,"mutex_wait_us":4,"spinlock_wait_cycles":3968,"thread_start_us":12361,"threads_started":1}
I20260929 01:12:04.652000 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.626463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39584 (local address 127.5.198.194:38947)
0929 01:12:04.641186 (+ 14723us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:04.641191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:04.641244 (+ 53us) server_negotiation.cc:407] Connection header received
0929 01:12:04.641357 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:04.641365 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:04.641612 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:04.641804 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:04.644181 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.645797 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.648525 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.649900 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.650053 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.650221 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:04.651509 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0929 01:12:04.651663 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14606,"thread_start_us":14499,"threads_started":1}
I20260929 01:12:04.652063 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.627375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.5.198.195:46189)
0929 01:12:04.640928 (+ 13553us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:04.640937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:04.641270 (+ 333us) server_negotiation.cc:407] Connection header received
0929 01:12:04.641390 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:04.641396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:04.641611 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:04.641804 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:04.644715 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.646264 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.648748 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.650135 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.650251 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.650479 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:04.651646 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0929 01:12:04.651760 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13374,"thread_start_us":13237,"threads_started":1}
I20260929 01:12:04.654541 23212 tablet_service.cc:1467] Processing CreateTablet for tablet 915cc42b81cf46f2a4f8fc820e4e8d15 (DEFAULT_TABLE table=test-table [id=394b7b8a4d33435893547449dfe4713f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:04.654896 23346 tablet_service.cc:1467] Processing CreateTablet for tablet 915cc42b81cf46f2a4f8fc820e4e8d15 (DEFAULT_TABLE table=test-table [id=394b7b8a4d33435893547449dfe4713f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:04.656288 23346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915cc42b81cf46f2a4f8fc820e4e8d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:04.658270 23212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915cc42b81cf46f2a4f8fc820e4e8d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:04.658316 23479 tablet_service.cc:1467] Processing CreateTablet for tablet 915cc42b81cf46f2a4f8fc820e4e8d15 (DEFAULT_TABLE table=test-table [id=394b7b8a4d33435893547449dfe4713f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:04.659675 23479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915cc42b81cf46f2a4f8fc820e4e8d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:04.668043 23562 tablet_bootstrap.cc:492] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a: Bootstrap starting.
I20260929 01:12:04.671204 23562 tablet_bootstrap.cc:654] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:04.672356 23562 log.cc:824] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:04.673707 23563 tablet_bootstrap.cc:492] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436: Bootstrap starting.
I20260929 01:12:04.675479 23562 tablet_bootstrap.cc:492] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a: No bootstrap required, opened a new log
I20260929 01:12:04.675751 23562 ts_tablet_manager.cc:1397] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 01:12:04.676514 23565 tablet_bootstrap.cc:492] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb: Bootstrap starting.
I20260929 01:12:04.677352 23563 tablet_bootstrap.cc:654] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:04.678565 23563 log.cc:824] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:04.679492 23565 tablet_bootstrap.cc:654] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:04.680640 23565 log.cc:824] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:04.681666 23563 tablet_bootstrap.cc:492] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436: No bootstrap required, opened a new log
I20260929 01:12:04.681968 23563 ts_tablet_manager.cc:1397] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260929 01:12:04.683574 23565 tablet_bootstrap.cc:492] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb: No bootstrap required, opened a new log
I20260929 01:12:04.683877 23565 ts_tablet_manager.cc:1397] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 01:12:04.686416 23562 raft_consensus.cc:348] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f14c884c9e4c16942f7323c020892a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38947 } } peers { permanent_uuid: "e16f7e9b588440409a6c662aa50d9436" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37461 } } peers { permanent_uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46189 } }
I20260929 01:12:04.686841 23562 raft_consensus.cc:374] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:04.687024 23562 raft_consensus.cc:729] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f14c884c9e4c16942f7323c020892a, State: Initialized, Role: FOLLOWER
I20260929 01:12:04.687640 23562 consensus_queue.cc:260] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f14c884c9e4c16942f7323c020892a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38947 } } peers { permanent_uuid: "e16f7e9b588440409a6c662aa50d9436" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37461 } } peers { permanent_uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46189 } }
I20260929 01:12:04.688853 23411 heartbeater.cc:503] Master 127.5.198.254:39261 was elected leader, sending a full tablet report...
I20260929 01:12:04.689414 23562 ts_tablet_manager.cc:1428] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260929 01:12:04.692195 23563 raft_consensus.cc:348] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f14c884c9e4c16942f7323c020892a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38947 } } peers { permanent_uuid: "e16f7e9b588440409a6c662aa50d9436" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37461 } } peers { permanent_uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46189 } }
I20260929 01:12:04.692721 23563 raft_consensus.cc:374] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:04.692865 23563 raft_consensus.cc:729] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16f7e9b588440409a6c662aa50d9436, State: Initialized, Role: FOLLOWER
I20260929 01:12:04.692888 23565 raft_consensus.cc:348] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f14c884c9e4c16942f7323c020892a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38947 } } peers { permanent_uuid: "e16f7e9b588440409a6c662aa50d9436" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37461 } } peers { permanent_uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46189 } }
I20260929 01:12:04.693364 23565 raft_consensus.cc:374] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:04.693518 23565 raft_consensus.cc:729] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b9f1a79d1f40c193c5023aa9f8ccfb, State: Initialized, Role: FOLLOWER
I20260929 01:12:04.693560 23563 consensus_queue.cc:260] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f14c884c9e4c16942f7323c020892a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38947 } } peers { permanent_uuid: "e16f7e9b588440409a6c662aa50d9436" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37461 } } peers { permanent_uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46189 } }
I20260929 01:12:04.694221 23565 consensus_queue.cc:260] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f14c884c9e4c16942f7323c020892a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38947 } } peers { permanent_uuid: "e16f7e9b588440409a6c662aa50d9436" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37461 } } peers { permanent_uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46189 } }
I20260929 01:12:04.695298 23563 ts_tablet_manager.cc:1428] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260929 01:12:04.695380 23544 heartbeater.cc:503] Master 127.5.198.254:39261 was elected leader, sending a full tablet report...
I20260929 01:12:04.696638 23565 ts_tablet_manager.cc:1428] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 01:12:04.726508 23570 raft_consensus.cc:482] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:12:04.726879 23570 raft_consensus.cc:504] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f14c884c9e4c16942f7323c020892a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38947 } } peers { permanent_uuid: "e16f7e9b588440409a6c662aa50d9436" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37461 } } peers { permanent_uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46189 } }
I20260929 01:12:04.728648 23570 leader_election.cc:288] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81f14c884c9e4c16942f7323c020892a (127.5.198.194:38947), e16f7e9b588440409a6c662aa50d9436 (127.5.198.193:37461)
I20260929 01:12:04.734309 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.728825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:38947 (local address 127.5.198.195:55537)
0929 01:12:04.729004 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:04.729045 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:04.729184 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:04.729504 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:04.729509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:04.729536 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:04.729757 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:04.729769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.730946 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.730961 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:04.732195 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:04.732206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.733572 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.733579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:04.733767 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.733792 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 01:12:04.733896 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 01:12:04.734027 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 01:12:04.734309 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.728974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:55537 (local address 127.5.198.194:38947)
0929 01:12:04.729178 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:04.729184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:04.729204 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:12:04.729307 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:04.729313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:04.729384 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:04.729481 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:04.729885 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.730812 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.732357 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.733351 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.733585 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.733660 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:04.733922 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 01:12:04.734052 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:12:04.735360 23561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.729539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:34489 (local address 127.5.198.193:37461)
0929 01:12:04.729779 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:04.729786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:04.730186 (+ 400us) server_negotiation.cc:407] Connection header received
0929 01:12:04.730300 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:04.730305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:04.730371 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:04.730452 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:04.730958 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.731663 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.733609 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.734467 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.734654 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.734725 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:04.734995 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 01:12:04.735120 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 01:12:04.735401 23571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.729269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37461 (local address 127.5.198.195:34489)
0929 01:12:04.730077 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:04.730091 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:04.730190 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:04.730498 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:04.730501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:04.730518 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:04.730809 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:04.730817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.731830 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.731833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:04.733427 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:04.733438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.734665 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.734671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:04.734833 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.734855 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:12:04.734954 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 01:12:04.735084 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":540,"threads_started":1}
I20260929 01:12:04.735244 23366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915cc42b81cf46f2a4f8fc820e4e8d15" candidate_uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f14c884c9e4c16942f7323c020892a" is_pre_election: true
I20260929 01:12:04.735780 23366 raft_consensus.cc:2454] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b9f1a79d1f40c193c5023aa9f8ccfb in term 0.
I20260929 01:12:04.736508 23433 leader_election.cc:302] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b9f1a79d1f40c193c5023aa9f8ccfb, 81f14c884c9e4c16942f7323c020892a; no voters:
I20260929 01:12:04.736354 23232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915cc42b81cf46f2a4f8fc820e4e8d15" candidate_uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16f7e9b588440409a6c662aa50d9436" is_pre_election: true
I20260929 01:12:04.736922 23570 raft_consensus.cc:2781] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:12:04.736970 23232 raft_consensus.cc:2454] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b9f1a79d1f40c193c5023aa9f8ccfb in term 0.
I20260929 01:12:04.737121 23570 raft_consensus.cc:482] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:12:04.737237 23570 raft_consensus.cc:3037] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:04.739439 23570 raft_consensus.cc:504] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f14c884c9e4c16942f7323c020892a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38947 } } peers { permanent_uuid: "e16f7e9b588440409a6c662aa50d9436" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37461 } } peers { permanent_uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46189 } }
I20260929 01:12:04.740020 23570 leader_election.cc:288] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [CANDIDATE]: Term 1 election: Requested vote from peers 81f14c884c9e4c16942f7323c020892a (127.5.198.194:38947), e16f7e9b588440409a6c662aa50d9436 (127.5.198.193:37461)
I20260929 01:12:04.740552 23366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915cc42b81cf46f2a4f8fc820e4e8d15" candidate_uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f14c884c9e4c16942f7323c020892a"
I20260929 01:12:04.740665 23232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915cc42b81cf46f2a4f8fc820e4e8d15" candidate_uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16f7e9b588440409a6c662aa50d9436"
I20260929 01:12:04.740760 23366 raft_consensus.cc:3037] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:04.740903 23232 raft_consensus.cc:3037] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:04.743438 23366 raft_consensus.cc:2454] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b9f1a79d1f40c193c5023aa9f8ccfb in term 1.
I20260929 01:12:04.743548 23232 raft_consensus.cc:2454] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b9f1a79d1f40c193c5023aa9f8ccfb in term 1.
I20260929 01:12:04.743826 23433 leader_election.cc:302] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b9f1a79d1f40c193c5023aa9f8ccfb, 81f14c884c9e4c16942f7323c020892a; no voters:
I20260929 01:12:04.744141 23570 raft_consensus.cc:2781] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:04.744642 23570 raft_consensus.cc:686] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [term 1 LEADER]: Becoming Leader. State: Replica: 20b9f1a79d1f40c193c5023aa9f8ccfb, State: Running, Role: LEADER
I20260929 01:12:04.745217 23570 consensus_queue.cc:237] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f14c884c9e4c16942f7323c020892a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38947 } } peers { permanent_uuid: "e16f7e9b588440409a6c662aa50d9436" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37461 } } peers { permanent_uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46189 } }
I20260929 01:12:04.749570 23087 catalog_manager.cc:5495] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b9f1a79d1f40c193c5023aa9f8ccfb (127.5.198.195). New cstate: current_term: 1 leader_uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f14c884c9e4c16942f7323c020892a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e16f7e9b588440409a6c662aa50d9436" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b9f1a79d1f40c193c5023aa9f8ccfb" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 46189 } health_report { overall_health: HEALTHY } } }
I20260929 01:12:04.778990 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.773705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.5.198.195:46189)
0929 01:12:04.773881 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:04.773885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:04.773905 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:12:04.774068 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:04.774072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:04.774174 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:04.774296 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:12:04.774677 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.775386 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.777080 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.777523 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.777623 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.778024 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:12:04.778520 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:12:04.778627 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:04.778734 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 01:12:04.778798 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 01:12:04.783505 23278 tablet.cc:2369] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:12:04.786864 23412 tablet.cc:2369] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:12:04.787251 23366 raft_consensus.cc:1261] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a [term 1 FOLLOWER]: Refusing update from remote peer 20b9f1a79d1f40c193c5023aa9f8ccfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:12:04.787355 23232 raft_consensus.cc:1261] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436 [term 1 FOLLOWER]: Refusing update from remote peer 20b9f1a79d1f40c193c5023aa9f8ccfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:12:04.788249 23570 consensus_queue.cc:1035] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "81f14c884c9e4c16942f7323c020892a" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 38947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:12:04.788622 23572 consensus_queue.cc:1035] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16f7e9b588440409a6c662aa50d9436" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:12:04.799611 23576 mvcc.cc:205] Tried to move back new op lower bound from 7334479154317250560 to 7334479154160689152. Current Snapshot: MvccSnapshot[applied={T|T < 7334479154317250560}]
I20260929 01:12:04.801496 23575 mvcc.cc:205] Tried to move back new op lower bound from 7334479154317250560 to 7334479154160689152. Current Snapshot: MvccSnapshot[applied={T|T < 7334479154317250560}]
I20260929 01:12:04.814211 23087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:12:04.814515 23087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:12:04.818266 23087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d951129c4cea4ffea6cd64fb591a8490: Sending DeleteTablet for 3 replicas of tablet 915cc42b81cf46f2a4f8fc820e4e8d15
I20260929 01:12:04.819428 23346 tablet_service.cc:1514] Processing DeleteTablet for tablet 915cc42b81cf46f2a4f8fc820e4e8d15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:04 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:39584
I20260929 01:12:04.819514 23212 tablet_service.cc:1514] Processing DeleteTablet for tablet 915cc42b81cf46f2a4f8fc820e4e8d15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:04 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:54362
I20260929 01:12:04.819834 23479 tablet_service.cc:1514] Processing DeleteTablet for tablet 915cc42b81cf46f2a4f8fc820e4e8d15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:04 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:54640
I20260929 01:12:04.821424 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 23149
I20260929 01:12:04.850886 23585 tablet_replica.cc:330] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436: stopping tablet replica
I20260929 01:12:04.853526 23585 raft_consensus.cc:2229] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:12:04.857064 23586 tablet_replica.cc:330] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb: stopping tablet replica
I20260929 01:12:04.857721 23584 tablet_replica.cc:330] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a: stopping tablet replica
I20260929 01:12:04.858217 23584 raft_consensus.cc:2229] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:12:04.858315 23585 raft_consensus.cc:2258] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:04.858705 23584 raft_consensus.cc:2258] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:04.858812 23586 raft_consensus.cc:2229] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:12:04.859498 23586 raft_consensus.cc:2258] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:04.861300 23584 ts_tablet_manager.cc:1905] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:04.862106 23586 ts_tablet_manager.cc:1905] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:04.862306 23585 ts_tablet_manager.cc:1905] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:04.868749 23584 ts_tablet_manager.cc:1918] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:12:04.868940 23584 log.cc:1196] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/915cc42b81cf46f2a4f8fc820e4e8d15
I20260929 01:12:04.869552 23584 ts_tablet_manager.cc:1939] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 81f14c884c9e4c16942f7323c020892a: Deleting consensus metadata
I20260929 01:12:04.871555 23073 catalog_manager.cc:4841] TS 81f14c884c9e4c16942f7323c020892a (127.5.198.194:38947): tablet 915cc42b81cf46f2a4f8fc820e4e8d15 (table test-table [id=394b7b8a4d33435893547449dfe4713f]) successfully deleted
I20260929 01:12:04.872759 23585 ts_tablet_manager.cc:1918] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:12:04.872920 23585 log.cc:1196] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/915cc42b81cf46f2a4f8fc820e4e8d15
Sep 29 01:12:04 dist-test-slave-41cl krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:12:04.873436 23585 ts_tablet_manager.cc:1939] T 915cc42b81cf46f2a4f8fc820e4e8d15 P e16f7e9b588440409a6c662aa50d9436: Deleting consensus metadata
I20260929 01:12:04.875311 23073 catalog_manager.cc:4841] TS e16f7e9b588440409a6c662aa50d9436 (127.5.198.193:37461): tablet 915cc42b81cf46f2a4f8fc820e4e8d15 (table test-table [id=394b7b8a4d33435893547449dfe4713f]) successfully deleted
I20260929 01:12:04.878973 23586 ts_tablet_manager.cc:1918] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:12:04.879258 23586 log.cc:1196] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/915cc42b81cf46f2a4f8fc820e4e8d15
I20260929 01:12:04.880008 23586 ts_tablet_manager.cc:1939] T 915cc42b81cf46f2a4f8fc820e4e8d15 P 20b9f1a79d1f40c193c5023aa9f8ccfb: Deleting consensus metadata
I20260929 01:12:04.882064 23072 catalog_manager.cc:4841] TS 20b9f1a79d1f40c193c5023aa9f8ccfb (127.5.198.195:46189): tablet 915cc42b81cf46f2a4f8fc820e4e8d15 (table test-table [id=394b7b8a4d33435893547449dfe4713f]) successfully deleted
I20260929 01:12:04.884492 23561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:04.823021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.5.198.193:37461)
0929 01:12:04.823260 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:04.823269 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:04.847029 (+ 23760us) server_negotiation.cc:407] Connection header received
0929 01:12:04.847727 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:04.847737 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:04.847864 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:04.847956 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:04.851222 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.852805 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.859861 (+ 7056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:04.860527 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:04.860640 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:04.876308 (+ 15668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:04.876325 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:04.876347 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:04.876402 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:04.880751 (+ 4349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:04.881908 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:04.881918 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:04.881921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:04.882098 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:04.882602 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:04.882607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:04.882610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:04.883121 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:04.883434 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:04.883822 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 01:12:04.884059 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 01:12:04.885519 23272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:04.987598 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 23149
I20260929 01:12:05.005767 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 23283
Sep 29 01:12:05 dist-test-slave-41cl krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:12:05.023376 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:05.006596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39590 (local address 127.5.198.194:38947)
0929 01:12:05.006806 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:05.006813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:05.006835 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:12:05.006959 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:05.006967 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:05.007052 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:05.007186 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:05.008506 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:05.009666 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:05.011127 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:05.011739 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:05.011873 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:05.016497 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:05.016517 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:05.016540 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:05.016584 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:05.020330 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:05.021200 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:05.021211 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:05.021215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:05.021364 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:05.021755 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:05.021760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:05.021763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:05.022238 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:05.022548 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:05.022768 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 01:12:05.022994 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:12:05.024133 23406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:05.113477 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 23283
I20260929 01:12:05.131129 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 23416
Sep 29 01:12:05 dist-test-slave-41cl krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:12:05.150138 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:05.132035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54656 (local address 127.5.198.195:46189)
0929 01:12:05.132308 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:05.132317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:05.132341 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:12:05.132420 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:05.132427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:05.132660 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:05.132792 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:05.134054 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:05.135398 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:05.136827 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:05.137611 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:05.137774 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:05.142585 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:05.142601 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:05.142622 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:05.142687 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:05.147047 (+ 4360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:05.147695 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:05.147706 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:05.147710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:05.147953 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:05.148320 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:05.148327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:05.148331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:05.148847 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:05.149227 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:05.149397 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 01:12:05.149639 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:12:05.151024 23539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:05.275131 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 23416
I20260929 01:12:05.294791 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 23056
I20260929 01:12:05.295972 23118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:05.407619 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 23056
2026-09-29T01:12:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4274 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T01:12:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:05Z Disabled control of system clock
I20260929 01:12:05.483326 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:42565
--webserver_interface=127.5.198.254
--webserver_port=0
--builtin_ntp_servers=127.5.198.212:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:42565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:12:05.812423 23595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:05.812788 23595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:05.812875 23595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:05.820648 23595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:05.820750 23595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:05.820786 23595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:05.820842 23595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:05.820884 23595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:05.830137 23595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:42565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:42565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:05.831362 23595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:05.833122 23595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:05.845080 23600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:05.845155 23603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:05.845208 23601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:05.845726 23595 server_base.cc:1034] running on GCE node
I20260929 01:12:05.846534 23595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:05.847860 23595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:05.849133 23595 hybrid_clock.cc:648] HybridClock initialized: now 1790644325849092 us; error 33 us; skew 500 ppm
I20260929 01:12:05.849486 23595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:12:05.865818 23595 webserver.cc:460] Webserver started at http://127.5.198.254:43385/ using document root <none> and password file <none>
I20260929 01:12:05.866580 23595 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:05.866683 23595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:05.866976 23595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:05.870142 23595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "5f974cb9f042407baf5a02494aec07f1"
format_stamp: "Formatted at 2026-09-29 01:12:05 on dist-test-slave-41cl"
server_key: "46eb56ed82efa3670f16a66f27b6a99d"
server_key_iv: "0e925159c2c1358d72c69a6d511df4f3"
server_key_version: "encryptionkey@0"
I20260929 01:12:05.870924 23595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "5f974cb9f042407baf5a02494aec07f1"
format_stamp: "Formatted at 2026-09-29 01:12:05 on dist-test-slave-41cl"
server_key: "46eb56ed82efa3670f16a66f27b6a99d"
server_key_iv: "0e925159c2c1358d72c69a6d511df4f3"
server_key_version: "encryptionkey@0"
I20260929 01:12:05.876446 23595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:12:05.880089 23609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:05.881652 23595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:12:05.881814 23595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "5f974cb9f042407baf5a02494aec07f1"
format_stamp: "Formatted at 2026-09-29 01:12:05 on dist-test-slave-41cl"
server_key: "46eb56ed82efa3670f16a66f27b6a99d"
server_key_iv: "0e925159c2c1358d72c69a6d511df4f3"
server_key_version: "encryptionkey@0"
I20260929 01:12:05.882073 23595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:05.907758 23595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:05.908607 23595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:05.908854 23595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:05.927958 23595 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:42565
I20260929 01:12:05.928066 23660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:42565 every 8 connection(s)
I20260929 01:12:05.929303 23595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:05.933058 23661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:05.935393 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 23595
I20260929 01:12:05.935586 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:05.936122 5915 external_mini_cluster.cc:1428] Setting key 6cc17cc7a8c5894d253c8c450d9c83b7
I20260929 01:12:05.945744 23661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1: Bootstrap starting.
I20260929 01:12:05.949457 23661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:05.950572 23661 log.cc:824] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:05.950861 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:05.938144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.5.198.254:42565)
0929 01:12:05.939203 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:05.939213 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:05.939254 (+ 41us) server_negotiation.cc:407] Connection header received
0929 01:12:05.940980 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:05.941003 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:05.941452 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:05.941770 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:05.942378 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:05.943810 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:05.945769 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:05.946686 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:05.946844 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:05.948247 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:05.948255 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:05.948268 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:05.948907 (+ 639us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:05.948994 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:05.949015 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:05.949116 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:05.949397 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 01:12:05.949762 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":391,"threads_started":1}
I20260929 01:12:05.953430 23661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1: No bootstrap required, opened a new log
I20260929 01:12:05.960930 23661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f974cb9f042407baf5a02494aec07f1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42565 } }
I20260929 01:12:05.961328 23661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:05.961434 23661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f974cb9f042407baf5a02494aec07f1, State: Initialized, Role: FOLLOWER
I20260929 01:12:05.962054 23661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f974cb9f042407baf5a02494aec07f1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42565 } }
I20260929 01:12:05.962298 23661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:05.962438 23661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:05.962617 23661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:05.965111 23661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f974cb9f042407baf5a02494aec07f1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42565 } }
I20260929 01:12:05.965602 23661 leader_election.cc:302] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f974cb9f042407baf5a02494aec07f1; no voters:
I20260929 01:12:05.966163 23661 leader_election.cc:288] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:05.966382 23666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:05.967788 23666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [term 1 LEADER]: Becoming Leader. State: Replica: 5f974cb9f042407baf5a02494aec07f1, State: Running, Role: LEADER
I20260929 01:12:05.968408 23661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:05.968343 23666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f974cb9f042407baf5a02494aec07f1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42565 } }
I20260929 01:12:05.971536 23668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f974cb9f042407baf5a02494aec07f1. Latest consensus state: current_term: 1 leader_uuid: "5f974cb9f042407baf5a02494aec07f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f974cb9f042407baf5a02494aec07f1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42565 } } }
I20260929 01:12:05.971948 23668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:05.972314 23667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f974cb9f042407baf5a02494aec07f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f974cb9f042407baf5a02494aec07f1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42565 } } }
I20260929 01:12:05.972689 23667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:05.974042 23675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:05.978873 23675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:05.988643 23675 catalog_manager.cc:1347] Generated new cluster ID: 3a0edb3419754a1498c615507b4d474e
I20260929 01:12:05.988755 23675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:05.997176 23675 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:05.998919 23675 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:06.018821 23675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1: Generated new TSK 0
I20260929 01:12:06.019950 23675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:06.025581 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42565
--builtin_ntp_servers=127.5.198.212:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:12:06.353031 23685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:06.353358 23685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:06.353520 23685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:06.360956 23685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:06.361088 23685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:06.361267 23685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:12:06.371301 23685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:06.372746 23685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:06.374467 23685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:06.387173 23691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:06.387483 23693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:06.387548 23690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:06.388018 23685 server_base.cc:1034] running on GCE node
I20260929 01:12:06.388695 23685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:06.389559 23685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:06.390771 23685 hybrid_clock.cc:648] HybridClock initialized: now 1790644326390728 us; error 42 us; skew 500 ppm
I20260929 01:12:06.391081 23685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:12:06.393455 23685 webserver.cc:460] Webserver started at http://127.5.198.193:38743/ using document root <none> and password file <none>
I20260929 01:12:06.394127 23685 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:06.394213 23685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:06.394521 23685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:06.397477 23685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "b2dc6470f1494144b146e73f5344fdb4"
format_stamp: "Formatted at 2026-09-29 01:12:06 on dist-test-slave-41cl"
server_key: "be31b5aa923f3175f0ef21d5bcbc7cc6"
server_key_iv: "de0a542eed4eb8a015758060cd5c5a4f"
server_key_version: "encryptionkey@0"
I20260929 01:12:06.398247 23685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "b2dc6470f1494144b146e73f5344fdb4"
format_stamp: "Formatted at 2026-09-29 01:12:06 on dist-test-slave-41cl"
server_key: "be31b5aa923f3175f0ef21d5bcbc7cc6"
server_key_iv: "de0a542eed4eb8a015758060cd5c5a4f"
server_key_version: "encryptionkey@0"
I20260929 01:12:06.403151 23685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260929 01:12:06.406565 23699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:06.408044 23685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:12:06.408226 23685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "b2dc6470f1494144b146e73f5344fdb4"
format_stamp: "Formatted at 2026-09-29 01:12:06 on dist-test-slave-41cl"
server_key: "be31b5aa923f3175f0ef21d5bcbc7cc6"
server_key_iv: "de0a542eed4eb8a015758060cd5c5a4f"
server_key_version: "encryptionkey@0"
I20260929 01:12:06.408435 23685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:06.423017 23685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:06.423789 23685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:06.424105 23685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:06.425128 23685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:06.426498 23685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:06.426622 23685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:06.426751 23685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:06.426837 23685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:06.462277 23685 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:39501
I20260929 01:12:06.462365 23811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:39501 every 8 connection(s)
I20260929 01:12:06.463593 23685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:12:06.468851 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 23685
I20260929 01:12:06.469033 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:12:06.469581 5915 external_mini_cluster.cc:1428] Setting key 941b9f80b8151b5fdac50bff969656ec
I20260929 01:12:06.472373 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42565
--builtin_ntp_servers=127.5.198.212:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:12:06.477988 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:06.466219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:40099 (local address 127.5.198.254:42565)
0929 01:12:06.466700 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:06.466705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:06.467715 (+ 1010us) server_negotiation.cc:407] Connection header received
0929 01:12:06.470113 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:06.470117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:06.470229 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:06.470358 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:06.471691 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:06.472675 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:06.475150 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:06.475558 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:06.475652 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:06.477247 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:06.477252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:06.477256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:06.477273 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:06.477323 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:06.477335 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:06.477410 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:06.477659 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 01:12:06.477732 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":262,"threads_started":1}
I20260929 01:12:06.479377 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:06.466713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:42565 (local address 127.5.198.193:40099)
0929 01:12:06.467561 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:06.467596 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:06.469807 (+ 2211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:06.470637 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:06.470649 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:06.470744 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:06.471529 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:06.471553 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:06.472809 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:06.472817 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:06.474993 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:06.475010 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:06.475775 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:06.475784 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:06.476108 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:06.476907 (+ 799us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:12:06.476913 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:06.476963 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:12:06.477065 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:06.477436 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:06.477448 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:06.477459 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 01:12:06.477640 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 01:12:06.478070 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":261,"threads_started":1}
I20260929 01:12:06.481079 23812 heartbeater.cc:347] Connected to a master server at 127.5.198.254:42565
I20260929 01:12:06.481416 23812 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:06.482355 23812 heartbeater.cc:511] Master 127.5.198.254:42565 requested a full tablet report, sending...
I20260929 01:12:06.484637 23626 ts_manager.cc:194] Registered new tserver with Master: b2dc6470f1494144b146e73f5344fdb4 (127.5.198.193:39501)
I20260929 01:12:06.486231 23626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.198.193:40099
W20260929 01:12:06.797154 23816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:06.797454 23816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:06.797578 23816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:06.805223 23816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:06.805351 23816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:06.805544 23816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:12:06.815452 23816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:06.816871 23816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:06.818645 23816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:06.832492 23821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:06.832407 23822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:06.832502 23824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:06.833083 23816 server_base.cc:1034] running on GCE node
I20260929 01:12:06.833741 23816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:06.834730 23816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:06.835958 23816 hybrid_clock.cc:648] HybridClock initialized: now 1790644326835919 us; error 57 us; skew 500 ppm
I20260929 01:12:06.836298 23816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:12:06.838550 23816 webserver.cc:460] Webserver started at http://127.5.198.194:44471/ using document root <none> and password file <none>
I20260929 01:12:06.839299 23816 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:06.839427 23816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:06.839746 23816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:06.842578 23816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "dd9c412435d84d81ad5032afb0cc8fea"
format_stamp: "Formatted at 2026-09-29 01:12:06 on dist-test-slave-41cl"
server_key: "3d4a20c7c8987af3e424b5898cf28f72"
server_key_iv: "460548772f865f672a23195e314a7545"
server_key_version: "encryptionkey@0"
I20260929 01:12:06.843355 23816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "dd9c412435d84d81ad5032afb0cc8fea"
format_stamp: "Formatted at 2026-09-29 01:12:06 on dist-test-slave-41cl"
server_key: "3d4a20c7c8987af3e424b5898cf28f72"
server_key_iv: "460548772f865f672a23195e314a7545"
server_key_version: "encryptionkey@0"
I20260929 01:12:06.848187 23816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260929 01:12:06.851568 23830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:06.852897 23816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:12:06.853087 23816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "dd9c412435d84d81ad5032afb0cc8fea"
format_stamp: "Formatted at 2026-09-29 01:12:06 on dist-test-slave-41cl"
server_key: "3d4a20c7c8987af3e424b5898cf28f72"
server_key_iv: "460548772f865f672a23195e314a7545"
server_key_version: "encryptionkey@0"
I20260929 01:12:06.853302 23816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:06.864171 23816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:06.865125 23816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:06.865458 23816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:06.866460 23816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:06.867945 23816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:06.868072 23816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:06.868214 23816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:06.868309 23816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:06.905428 23816 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:36791
I20260929 01:12:06.905480 23942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:36791 every 8 connection(s)
I20260929 01:12:06.907009 23816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:12:06.910647 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 23816
I20260929 01:12:06.910841 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:12:06.911286 5915 external_mini_cluster.cc:1428] Setting key 17600aede2b250d9ce0e9fa3a6d8a558
I20260929 01:12:06.914773 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42565
--builtin_ntp_servers=127.5.198.212:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:12:06.921736 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:06.909734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:37747 (local address 127.5.198.254:42565)
0929 01:12:06.909946 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:06.909952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:06.911624 (+ 1672us) server_negotiation.cc:407] Connection header received
0929 01:12:06.914024 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:06.914030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:06.914118 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:06.914241 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:06.915537 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:06.916793 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:06.918361 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:06.919124 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:06.919279 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:06.920713 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:06.920723 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:06.920732 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:06.920760 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:06.920810 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:06.920831 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:06.920940 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:06.921142 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 01:12:06.921266 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:12:06.922458 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:06.910334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:42565 (local address 127.5.198.194:37747)
0929 01:12:06.911445 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:06.911491 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:06.913754 (+ 2263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:06.914446 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:06.914454 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:06.914547 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:06.915351 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:06.915369 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:06.916954 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:06.916957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:06.918204 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:06.918216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:06.919355 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:06.919359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:06.919563 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:06.920324 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:12:06.920326 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:06.920371 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:12:06.920449 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:06.920934 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:06.920940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:06.920946 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 01:12:06.921100 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 01:12:06.921392 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":331,"threads_started":1}
I20260929 01:12:06.923928 23943 heartbeater.cc:347] Connected to a master server at 127.5.198.254:42565
I20260929 01:12:06.924292 23943 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:06.925251 23943 heartbeater.cc:511] Master 127.5.198.254:42565 requested a full tablet report, sending...
I20260929 01:12:06.926949 23626 ts_manager.cc:194] Registered new tserver with Master: dd9c412435d84d81ad5032afb0cc8fea (127.5.198.194:36791)
I20260929 01:12:06.928126 23626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.198.194:37747
W20260929 01:12:07.248446 23946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:07.248858 23946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:07.249035 23946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:07.257531 23946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:07.257663 23946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:07.257869 23946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:12:07.267648 23946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:07.269063 23946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:07.270862 23946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:07.284898 23951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:07.284940 23952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:07.285373 23954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:07.291517 23946 server_base.cc:1034] running on GCE node
I20260929 01:12:07.292162 23946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:07.293215 23946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:07.294447 23946 hybrid_clock.cc:648] HybridClock initialized: now 1790644327294406 us; error 56 us; skew 500 ppm
I20260929 01:12:07.294768 23946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:12:07.297158 23946 webserver.cc:460] Webserver started at http://127.5.198.195:38307/ using document root <none> and password file <none>
I20260929 01:12:07.297878 23946 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:07.298004 23946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:07.298353 23946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:07.301440 23946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "5ed9e66dfe88499a8a4c148c6b373315"
format_stamp: "Formatted at 2026-09-29 01:12:07 on dist-test-slave-41cl"
server_key: "98a2de531cc2032078f88bbaf27518c3"
server_key_iv: "64714f6a815eb22dd9ad7d1cfec01b04"
server_key_version: "encryptionkey@0"
I20260929 01:12:07.302244 23946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "5ed9e66dfe88499a8a4c148c6b373315"
format_stamp: "Formatted at 2026-09-29 01:12:07 on dist-test-slave-41cl"
server_key: "98a2de531cc2032078f88bbaf27518c3"
server_key_iv: "64714f6a815eb22dd9ad7d1cfec01b04"
server_key_version: "encryptionkey@0"
I20260929 01:12:07.307421 23946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 01:12:07.311184 23960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:07.312768 23946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:12:07.312956 23946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "5ed9e66dfe88499a8a4c148c6b373315"
format_stamp: "Formatted at 2026-09-29 01:12:07 on dist-test-slave-41cl"
server_key: "98a2de531cc2032078f88bbaf27518c3"
server_key_iv: "64714f6a815eb22dd9ad7d1cfec01b04"
server_key_version: "encryptionkey@0"
I20260929 01:12:07.313225 23946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:07.344631 23946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:07.345440 23946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:07.345736 23946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:07.346731 23946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:07.348168 23946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:07.348281 23946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:07.348408 23946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:07.348508 23946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:07.383824 23946 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:37493
I20260929 01:12:07.383913 24072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:37493 every 8 connection(s)
I20260929 01:12:07.385241 23946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:12:07.391825 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 23946
I20260929 01:12:07.392058 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:12:07.392572 5915 external_mini_cluster.cc:1428] Setting key b288f47936e8290a52d2a190d85f32e9
I20260929 01:12:07.398605 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.387943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:36283 (local address 127.5.198.254:42565)
0929 01:12:07.388218 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:07.388222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:07.389590 (+ 1368us) server_negotiation.cc:407] Connection header received
0929 01:12:07.391437 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:07.391442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:07.391542 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:07.391656 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:07.393035 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.394078 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.395872 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.396306 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.396445 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.397778 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:07.397783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:07.397787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:07.397803 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:07.397835 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:07.397846 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:07.397981 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:07.398191 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 01:12:07.398287 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:12:07.399582 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.388457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:42565 (local address 127.5.198.195:36283)
0929 01:12:07.389398 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:07.389433 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:07.391156 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:07.391909 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:07.391918 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:07.392023 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:07.392830 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:07.392858 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.394232 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.394238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:07.395725 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:07.395737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.396575 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.396586 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:07.396806 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.397461 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:12:07.397464 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:07.397503 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:12:07.397577 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:07.397944 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:07.397960 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:07.397967 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 01:12:07.398165 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 01:12:07.398552 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":252,"threads_started":1}
I20260929 01:12:07.401144 24073 heartbeater.cc:347] Connected to a master server at 127.5.198.254:42565
I20260929 01:12:07.401479 24073 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:07.402215 24073 heartbeater.cc:511] Master 127.5.198.254:42565 requested a full tablet report, sending...
I20260929 01:12:07.403554 23626 ts_manager.cc:194] Registered new tserver with Master: 5ed9e66dfe88499a8a4c148c6b373315 (127.5.198.195:37493)
I20260929 01:12:07.404378 23626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.198.195:36283
I20260929 01:12:07.407835 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:12:07.418272 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.412014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.5.198.254:42565)
0929 01:12:07.412281 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:07.412291 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:07.412751 (+ 460us) server_negotiation.cc:407] Connection header received
0929 01:12:07.412819 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:07.412830 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:07.412902 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:07.413003 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:07.413538 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.414315 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.415635 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.416113 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.416208 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.417371 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:07.417376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:07.417379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:07.417399 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:07.417447 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:07.417460 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:07.417539 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:07.417769 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 01:12:07.417888 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 01:12:07.421944 23626 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:12:07.428767 23626 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:12:07.464708 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.441837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39501 (local address 127.0.0.1:42718)
0929 01:12:07.455746 (+ 13909us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:07.455772 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:07.455895 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:07.457451 (+ 1556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:07.457456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:07.457480 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:07.457842 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:07.457854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.459513 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.459518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:07.461640 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:07.461653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.463891 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.463900 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:07.464087 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.464119 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 01:12:07.464205 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 01:12:07.464329 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13771,"thread_start_us":14500,"threads_started":1}
I20260929 01:12:07.465751 24085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.440341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:37493 (local address 127.0.0.1:54184)
0929 01:12:07.455068 (+ 14727us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:07.455098 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:07.455243 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:07.458612 (+ 3369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:07.458619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:07.458650 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:07.459060 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:07.459072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.460887 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.460892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:07.463051 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:07.463064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.464868 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.464876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:07.465064 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.465095 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 01:12:07.465165 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 01:12:07.465299 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14540,"mutex_wait_us":1,"thread_start_us":10444,"threads_started":1}
I20260929 01:12:07.465790 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.442152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42718 (local address 127.5.198.193:39501)
0929 01:12:07.456494 (+ 14342us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:07.456506 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:07.456538 (+ 32us) server_negotiation.cc:407] Connection header received
0929 01:12:07.456741 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:07.456752 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:07.457000 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:07.457233 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:07.458011 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.459373 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.461879 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.463390 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.463510 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.463768 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:07.465233 (+ 1465us) server_negotiation.cc:299] Negotiation successful
0929 01:12:07.465405 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14123,"thread_start_us":13997,"threads_started":1}
I20260929 01:12:07.465950 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.440277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.5.198.194:36791)
0929 01:12:07.454680 (+ 14403us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:07.454692 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:07.454727 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:12:07.454827 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:07.454834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:07.455160 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:07.455391 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:07.456632 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.458310 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.461242 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.462791 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.462915 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.463184 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:07.465395 (+ 2211us) server_negotiation.cc:299] Negotiation successful
0929 01:12:07.465581 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14196,"thread_start_us":14087,"threads_started":1}
I20260929 01:12:07.466450 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.440073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:36791 (local address 127.0.0.1:36838)
0929 01:12:07.451370 (+ 11297us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:07.451432 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:07.451641 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:07.455742 (+ 4101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:07.455753 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:07.455797 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:07.456203 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:07.456213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.458434 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.458439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:07.460975 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:07.460990 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.463312 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.463319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:07.463522 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.463606 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 01:12:07.463761 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 01:12:07.463958 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10873,"thread_start_us":14653,"threads_started":1}
I20260929 01:12:07.468359 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.441123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.5.198.195:37493)
0929 01:12:07.458075 (+ 16952us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:07.458092 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:07.458130 (+ 38us) server_negotiation.cc:407] Connection header received
0929 01:12:07.458222 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:07.458230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:07.458429 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:07.458612 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:07.459267 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.460685 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.463287 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.464654 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.466710 (+ 2056us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.467691 (+ 981us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:07.467768 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 01:12:07.467925 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16752,"thread_start_us":16506,"threads_started":1}
I20260929 01:12:07.469673 23747 tablet_service.cc:1467] Processing CreateTablet for tablet 407df744c0534fe79d0739854ec5fa52 (DEFAULT_TABLE table=test-table [id=80adf042bcbb44ea82fde5536197663f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:07.470089 23878 tablet_service.cc:1467] Processing CreateTablet for tablet 407df744c0534fe79d0739854ec5fa52 (DEFAULT_TABLE table=test-table [id=80adf042bcbb44ea82fde5536197663f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:07.471261 23747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407df744c0534fe79d0739854ec5fa52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:07.471271 23878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407df744c0534fe79d0739854ec5fa52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:07.471925 24008 tablet_service.cc:1467] Processing CreateTablet for tablet 407df744c0534fe79d0739854ec5fa52 (DEFAULT_TABLE table=test-table [id=80adf042bcbb44ea82fde5536197663f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:07.473299 24008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407df744c0534fe79d0739854ec5fa52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:07.483688 24091 tablet_bootstrap.cc:492] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea: Bootstrap starting.
I20260929 01:12:07.485397 24092 tablet_bootstrap.cc:492] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4: Bootstrap starting.
I20260929 01:12:07.487170 24093 tablet_bootstrap.cc:492] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315: Bootstrap starting.
I20260929 01:12:07.487350 24091 tablet_bootstrap.cc:654] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:07.488628 24091 log.cc:824] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:07.489008 24092 tablet_bootstrap.cc:654] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:07.489781 23812 heartbeater.cc:503] Master 127.5.198.254:42565 was elected leader, sending a full tablet report...
I20260929 01:12:07.490236 24093 tablet_bootstrap.cc:654] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:07.490329 24092 log.cc:824] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:07.491250 24093 log.cc:824] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:07.493404 24091 tablet_bootstrap.cc:492] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea: No bootstrap required, opened a new log
I20260929 01:12:07.493616 24092 tablet_bootstrap.cc:492] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4: No bootstrap required, opened a new log
I20260929 01:12:07.493768 24091 ts_tablet_manager.cc:1397] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 01:12:07.493990 24092 ts_tablet_manager.cc:1397] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 01:12:07.497753 24093 tablet_bootstrap.cc:492] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315: No bootstrap required, opened a new log
I20260929 01:12:07.498015 24093 ts_tablet_manager.cc:1397] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260929 01:12:07.501556 24091 raft_consensus.cc:348] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9c412435d84d81ad5032afb0cc8fea" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 36791 } } peers { permanent_uuid: "5ed9e66dfe88499a8a4c148c6b373315" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37493 } } peers { permanent_uuid: "b2dc6470f1494144b146e73f5344fdb4" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39501 } }
I20260929 01:12:07.501909 24091 raft_consensus.cc:374] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:07.502008 24091 raft_consensus.cc:729] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd9c412435d84d81ad5032afb0cc8fea, State: Initialized, Role: FOLLOWER
I20260929 01:12:07.502617 24091 consensus_queue.cc:260] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9c412435d84d81ad5032afb0cc8fea" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 36791 } } peers { permanent_uuid: "5ed9e66dfe88499a8a4c148c6b373315" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37493 } } peers { permanent_uuid: "b2dc6470f1494144b146e73f5344fdb4" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39501 } }
I20260929 01:12:07.503696 23943 heartbeater.cc:503] Master 127.5.198.254:42565 was elected leader, sending a full tablet report...
I20260929 01:12:07.504375 24091 ts_tablet_manager.cc:1428] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260929 01:12:07.504266 24092 raft_consensus.cc:348] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9c412435d84d81ad5032afb0cc8fea" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 36791 } } peers { permanent_uuid: "5ed9e66dfe88499a8a4c148c6b373315" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37493 } } peers { permanent_uuid: "b2dc6470f1494144b146e73f5344fdb4" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39501 } }
I20260929 01:12:07.504693 24092 raft_consensus.cc:374] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:07.504773 24092 raft_consensus.cc:729] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2dc6470f1494144b146e73f5344fdb4, State: Initialized, Role: FOLLOWER
I20260929 01:12:07.505918 24092 consensus_queue.cc:260] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9c412435d84d81ad5032afb0cc8fea" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 36791 } } peers { permanent_uuid: "5ed9e66dfe88499a8a4c148c6b373315" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37493 } } peers { permanent_uuid: "b2dc6470f1494144b146e73f5344fdb4" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39501 } }
I20260929 01:12:07.508275 24092 ts_tablet_manager.cc:1428] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260929 01:12:07.508090 24093 raft_consensus.cc:348] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9c412435d84d81ad5032afb0cc8fea" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 36791 } } peers { permanent_uuid: "5ed9e66dfe88499a8a4c148c6b373315" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37493 } } peers { permanent_uuid: "b2dc6470f1494144b146e73f5344fdb4" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39501 } }
I20260929 01:12:07.508545 24093 raft_consensus.cc:374] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:07.508692 24093 raft_consensus.cc:729] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed9e66dfe88499a8a4c148c6b373315, State: Initialized, Role: FOLLOWER
I20260929 01:12:07.509249 24093 consensus_queue.cc:260] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9c412435d84d81ad5032afb0cc8fea" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 36791 } } peers { permanent_uuid: "5ed9e66dfe88499a8a4c148c6b373315" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37493 } } peers { permanent_uuid: "b2dc6470f1494144b146e73f5344fdb4" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39501 } }
I20260929 01:12:07.510354 24073 heartbeater.cc:503] Master 127.5.198.254:42565 was elected leader, sending a full tablet report...
I20260929 01:12:07.510862 24093 ts_tablet_manager.cc:1428] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 01:12:07.556910 24098 raft_consensus.cc:482] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:12:07.557267 24098 raft_consensus.cc:504] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9c412435d84d81ad5032afb0cc8fea" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 36791 } } peers { permanent_uuid: "5ed9e66dfe88499a8a4c148c6b373315" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37493 } } peers { permanent_uuid: "b2dc6470f1494144b146e73f5344fdb4" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39501 } }
I20260929 01:12:07.559044 24098 leader_election.cc:288] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd9c412435d84d81ad5032afb0cc8fea (127.5.198.194:36791), 5ed9e66dfe88499a8a4c148c6b373315 (127.5.198.195:37493)
I20260929 01:12:07.565804 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.559444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:56559 (local address 127.5.198.194:36791)
0929 01:12:07.559686 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:07.559695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:07.560375 (+ 680us) server_negotiation.cc:407] Connection header received
0929 01:12:07.560640 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:07.560650 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:07.560718 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:07.560815 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:07.561437 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.562142 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.564106 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.564885 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.565043 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.565126 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:07.565405 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 01:12:07.565526 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:12:07.565840 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.559677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:48189 (local address 127.5.198.195:37493)
0929 01:12:07.559865 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:07.559872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:07.560409 (+ 537us) server_negotiation.cc:407] Connection header received
0929 01:12:07.560639 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:07.560649 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:07.560708 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:07.560802 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:07.561512 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.562224 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.564076 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.564926 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.565107 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.565175 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:07.565457 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 01:12:07.565600 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 01:12:07.565950 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.559421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:36791 (local address 127.5.198.193:56559)
0929 01:12:07.560271 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:07.560287 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:07.560447 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:07.560830 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:07.560834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:07.560856 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:07.561293 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:07.561305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.562281 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.562286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:07.563893 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:07.563905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.565073 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.565079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:07.565234 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.565256 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:12:07.565380 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 01:12:07.565506 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":103,"thread_start_us":563,"threads_started":1}
I20260929 01:12:07.566255 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.559439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:37493 (local address 127.5.198.193:48189)
0929 01:12:07.560308 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:07.560322 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:07.560466 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:07.560817 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:07.560821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:07.560844 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:07.561350 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:07.561360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.562353 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.562359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:07.563888 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:07.563900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.565114 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.565121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:07.565289 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.565311 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:12:07.565433 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 01:12:07.565558 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":81,"thread_start_us":472,"threads_started":1}
I20260929 01:12:07.566944 23898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407df744c0534fe79d0739854ec5fa52" candidate_uuid: "b2dc6470f1494144b146e73f5344fdb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd9c412435d84d81ad5032afb0cc8fea" is_pre_election: true
I20260929 01:12:07.567142 24028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407df744c0534fe79d0739854ec5fa52" candidate_uuid: "b2dc6470f1494144b146e73f5344fdb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed9e66dfe88499a8a4c148c6b373315" is_pre_election: true
I20260929 01:12:07.567456 23898 raft_consensus.cc:2454] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2dc6470f1494144b146e73f5344fdb4 in term 0.
I20260929 01:12:07.567669 24028 raft_consensus.cc:2454] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2dc6470f1494144b146e73f5344fdb4 in term 0.
I20260929 01:12:07.568120 23701 leader_election.cc:302] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2dc6470f1494144b146e73f5344fdb4, dd9c412435d84d81ad5032afb0cc8fea; no voters:
I20260929 01:12:07.568563 24098 raft_consensus.cc:2781] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:12:07.568727 24098 raft_consensus.cc:482] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:12:07.568877 24098 raft_consensus.cc:3037] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:07.571164 24098 raft_consensus.cc:504] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9c412435d84d81ad5032afb0cc8fea" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 36791 } } peers { permanent_uuid: "5ed9e66dfe88499a8a4c148c6b373315" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37493 } } peers { permanent_uuid: "b2dc6470f1494144b146e73f5344fdb4" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39501 } }
I20260929 01:12:07.571849 24098 leader_election.cc:288] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [CANDIDATE]: Term 1 election: Requested vote from peers dd9c412435d84d81ad5032afb0cc8fea (127.5.198.194:36791), 5ed9e66dfe88499a8a4c148c6b373315 (127.5.198.195:37493)
I20260929 01:12:07.572196 23898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407df744c0534fe79d0739854ec5fa52" candidate_uuid: "b2dc6470f1494144b146e73f5344fdb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd9c412435d84d81ad5032afb0cc8fea"
I20260929 01:12:07.572324 24028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407df744c0534fe79d0739854ec5fa52" candidate_uuid: "b2dc6470f1494144b146e73f5344fdb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed9e66dfe88499a8a4c148c6b373315"
I20260929 01:12:07.572394 23898 raft_consensus.cc:3037] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:07.572570 24028 raft_consensus.cc:3037] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:07.574673 23898 raft_consensus.cc:2454] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2dc6470f1494144b146e73f5344fdb4 in term 1.
I20260929 01:12:07.574702 24028 raft_consensus.cc:2454] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2dc6470f1494144b146e73f5344fdb4 in term 1.
I20260929 01:12:07.575093 23702 leader_election.cc:302] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5ed9e66dfe88499a8a4c148c6b373315, b2dc6470f1494144b146e73f5344fdb4, dd9c412435d84d81ad5032afb0cc8fea; no voters:
I20260929 01:12:07.575444 24098 raft_consensus.cc:2781] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:07.575891 24098 raft_consensus.cc:686] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [term 1 LEADER]: Becoming Leader. State: Replica: b2dc6470f1494144b146e73f5344fdb4, State: Running, Role: LEADER
I20260929 01:12:07.576334 24098 consensus_queue.cc:237] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9c412435d84d81ad5032afb0cc8fea" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 36791 } } peers { permanent_uuid: "5ed9e66dfe88499a8a4c148c6b373315" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37493 } } peers { permanent_uuid: "b2dc6470f1494144b146e73f5344fdb4" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39501 } }
I20260929 01:12:07.580920 23626 catalog_manager.cc:5495] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2dc6470f1494144b146e73f5344fdb4 (127.5.198.193). New cstate: current_term: 1 leader_uuid: "b2dc6470f1494144b146e73f5344fdb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd9c412435d84d81ad5032afb0cc8fea" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 36791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed9e66dfe88499a8a4c148c6b373315" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2dc6470f1494144b146e73f5344fdb4" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39501 } health_report { overall_health: HEALTHY } } }
I20260929 01:12:07.621563 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.615231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.5.198.193:39501)
0929 01:12:07.615415 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:07.615419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:07.615788 (+ 369us) server_negotiation.cc:407] Connection header received
0929 01:12:07.615896 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:07.615900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:07.616066 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:07.616231 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:12:07.616774 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.617772 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.619272 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.619699 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.619809 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.620400 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:12:07.621036 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:12:07.621155 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:07.621288 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 01:12:07.621356 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:12:07.630160 23898 raft_consensus.cc:1261] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea [term 1 FOLLOWER]: Refusing update from remote peer b2dc6470f1494144b146e73f5344fdb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:12:07.630174 24028 raft_consensus.cc:1261] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315 [term 1 FOLLOWER]: Refusing update from remote peer b2dc6470f1494144b146e73f5344fdb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:12:07.631140 24102 consensus_queue.cc:1035] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd9c412435d84d81ad5032afb0cc8fea" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:12:07.631536 24098 consensus_queue.cc:1035] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed9e66dfe88499a8a4c148c6b373315" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 37493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 01:12:07.636987 24074 tablet.cc:2369] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:12:07.644076 24105 mvcc.cc:205] Tried to move back new op lower bound from 7334479165962637312 to 7334479165756387328. Current Snapshot: MvccSnapshot[applied={T|T < 7334479165962637312}]
I20260929 01:12:07.656018 23625 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:12:07.656351 23625 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:12:07.660202 23625 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f974cb9f042407baf5a02494aec07f1: Sending DeleteTablet for 3 replicas of tablet 407df744c0534fe79d0739854ec5fa52
I20260929 01:12:07.661240 23878 tablet_service.cc:1514] Processing DeleteTablet for tablet 407df744c0534fe79d0739854ec5fa52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:07 UTC) from {username='slave'} at 127.0.0.1:36838
I20260929 01:12:07.661407 24008 tablet_service.cc:1514] Processing DeleteTablet for tablet 407df744c0534fe79d0739854ec5fa52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:07 UTC) from {username='slave'} at 127.0.0.1:54184
I20260929 01:12:07.661646 23747 tablet_service.cc:1514] Processing DeleteTablet for tablet 407df744c0534fe79d0739854ec5fa52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:07 UTC) from {username='slave'} at 127.0.0.1:42718
I20260929 01:12:07.663460 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 23685
I20260929 01:12:07.690495 24117 tablet_replica.cc:330] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4: stopping tablet replica
I20260929 01:12:07.696818 24117 raft_consensus.cc:2229] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:12:07.697830 24117 raft_consensus.cc:2258] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:07.698868 24115 tablet_replica.cc:330] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea: stopping tablet replica
I20260929 01:12:07.700901 24116 tablet_replica.cc:330] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315: stopping tablet replica
I20260929 01:12:07.700870 24115 raft_consensus.cc:2229] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:12:07.701283 24116 raft_consensus.cc:2229] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:12:07.701835 24115 raft_consensus.cc:2258] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:07.701835 24116 raft_consensus.cc:2258] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:07.702234 24117 ts_tablet_manager.cc:1905] T 407df744c0534fe79d0739854ec5fa52 P b2dc6470f1494144b146e73f5344fdb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:07.703176 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.665109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.5.198.193:39501)
0929 01:12:07.665319 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:07.665327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:07.689309 (+ 23982us) server_negotiation.cc:407] Connection header received
0929 01:12:07.690819 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:07.690829 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:07.690976 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:07.691081 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:07.691989 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.693637 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.698832 (+ 5195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.699528 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.699625 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.701482 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:07.701495 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:07.701513 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:07.702266 (+ 753us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:07.702327 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:07.702358 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:07.702458 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:07.702660 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 01:12:07.702758 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:12:07.703934 24116 ts_tablet_manager.cc:1905] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:07.704321 23807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:07.706404 24115 ts_tablet_manager.cc:1905] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:07.713004 24115 ts_tablet_manager.cc:1918] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:12:07.713173 24115 log.cc:1196] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/407df744c0534fe79d0739854ec5fa52
I20260929 01:12:07.713620 24115 ts_tablet_manager.cc:1939] T 407df744c0534fe79d0739854ec5fa52 P dd9c412435d84d81ad5032afb0cc8fea: Deleting consensus metadata
I20260929 01:12:07.713730 24116 ts_tablet_manager.cc:1918] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:12:07.713891 24116 log.cc:1196] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/407df744c0534fe79d0739854ec5fa52
I20260929 01:12:07.714396 24116 ts_tablet_manager.cc:1939] T 407df744c0534fe79d0739854ec5fa52 P 5ed9e66dfe88499a8a4c148c6b373315: Deleting consensus metadata
I20260929 01:12:07.715365 23611 catalog_manager.cc:4841] TS dd9c412435d84d81ad5032afb0cc8fea (127.5.198.194:36791): tablet 407df744c0534fe79d0739854ec5fa52 (table test-table [id=80adf042bcbb44ea82fde5536197663f]) successfully deleted
I20260929 01:12:07.716038 23612 catalog_manager.cc:4841] TS 5ed9e66dfe88499a8a4c148c6b373315 (127.5.198.195:37493): tablet 407df744c0534fe79d0739854ec5fa52 (table test-table [id=80adf042bcbb44ea82fde5536197663f]) successfully deleted
I20260929 01:12:07.804893 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 23685
W20260929 01:12:07.823468 23613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.198.193:39501 (error 108)
I20260929 01:12:07.823622 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 23816
I20260929 01:12:07.825376 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.824932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39501 (local address 127.0.0.1:42742)
0929 01:12:07.825117 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:07.825242 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 01:12:07.825625 23613 catalog_manager.cc:4584] TS b2dc6470f1494144b146e73f5344fdb4 (127.5.198.193:39501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 407df744c0534fe79d0739854ec5fa52: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39501: connect: Connection refused (error 111)
I20260929 01:12:07.830693 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.824543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.5.198.194:36791)
0929 01:12:07.824754 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:07.824760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:07.824782 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:12:07.824854 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:07.824859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:07.824987 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:07.825093 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:07.825519 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.826329 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.827609 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.828001 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.828135 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.829332 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:07.829341 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:07.829353 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:07.829900 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:07.829944 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:07.829966 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:07.830112 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:07.830270 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 01:12:07.830369 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 01:12:07.831359 23938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:07.879238 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.878722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39501 (local address 127.0.0.1:42746)
0929 01:12:07.878941 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:07.879063 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 01:12:07.880327 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.879945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39501 (local address 127.0.0.1:42758)
0929 01:12:07.880140 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:07.880207 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 01:12:07.926239 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 23816
I20260929 01:12:07.941174 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.940619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39501 (local address 127.0.0.1:42774)
0929 01:12:07.940859 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:07.940946 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 01:12:07.942251 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.941897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39501 (local address 127.0.0.1:42782)
0929 01:12:07.942048 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:07.942126 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 01:12:07.950052 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 23946
I20260929 01:12:07.957455 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:07.950876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54186 (local address 127.5.198.195:37493)
0929 01:12:07.951072 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:07.951077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:07.951107 (+ 30us) server_negotiation.cc:407] Connection header received
0929 01:12:07.951303 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:07.951309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:07.951408 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:07.951517 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:07.951941 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.952763 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.954088 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:07.954477 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:07.954613 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:07.955881 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:07.955890 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:07.955905 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:07.956616 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:07.956699 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:07.956726 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:07.956859 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:07.957042 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 01:12:07.957156 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:12:07.958149 24068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:08.023072 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:08.022520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39501 (local address 127.0.0.1:42784)
0929 01:12:08.022773 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:08.022887 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 01:12:08.024212 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:08.023769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39501 (local address 127.0.0.1:42790)
0929 01:12:08.023987 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:08.024074 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 01:12:08.046875 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 23946
I20260929 01:12:08.063895 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 23595
I20260929 01:12:08.065177 23656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:08.176745 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 23595
2026-09-29T01:12:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2771 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:08 dist-test-slave-41cl krb5kdc[24126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:08 dist-test-slave-41cl krb5kdc[24126](info): set up 2 sockets
Sep 29 01:12:08 dist-test-slave-41cl krb5kdc[24126](info): commencing operation
krb5kdc: starting...
W20260929 01:12:09.296633 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:09 dist-test-slave-41cl krb5kdc[24126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:12:09.545367 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:33345
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:33345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:09.880694 24139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:09.881048 24139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:09.881201 24139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:09.888674 24139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:09.888819 24139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:09.888933 24139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:09.889030 24139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:09.889117 24139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:09.899055 24139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:33345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:33345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:09.900417 24139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:09.901782 24139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:09.913923 24144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:09.913882 24147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:09.913944 24145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:09.914449 24139 server_base.cc:1034] running on GCE node
I20260929 01:12:09.915305 24139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:09.916811 24139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:09.918095 24139 hybrid_clock.cc:648] HybridClock initialized: now 1790644329918054 us; error 52 us; skew 500 ppm
I20260929 01:12:09.918462 24139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:09 dist-test-slave-41cl krb5kdc[24126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644329, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:09.921875 24139 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:12:09.923488 24139 webserver.cc:460] Webserver started at http://127.5.198.254:45287/ using document root <none> and password file <none>
I20260929 01:12:09.924182 24139 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:09.924285 24139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:09.924674 24139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:09.927690 24139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "a4726d0d69274fa2bcf7f0739d9c2b43"
format_stamp: "Formatted at 2026-09-29 01:12:09 on dist-test-slave-41cl"
server_key: "a3567f720afe76ea4272bb8cd9c19ca6"
server_key_iv: "f0127ef47438976c0449d3a43ca6bb44"
server_key_version: "encryptionkey@0"
I20260929 01:12:09.928452 24139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "a4726d0d69274fa2bcf7f0739d9c2b43"
format_stamp: "Formatted at 2026-09-29 01:12:09 on dist-test-slave-41cl"
server_key: "a3567f720afe76ea4272bb8cd9c19ca6"
server_key_iv: "f0127ef47438976c0449d3a43ca6bb44"
server_key_version: "encryptionkey@0"
I20260929 01:12:09.933769 24139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 01:12:09.937327 24154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:09.938709 24139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:12:09.938907 24139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "a4726d0d69274fa2bcf7f0739d9c2b43"
format_stamp: "Formatted at 2026-09-29 01:12:09 on dist-test-slave-41cl"
server_key: "a3567f720afe76ea4272bb8cd9c19ca6"
server_key_iv: "f0127ef47438976c0449d3a43ca6bb44"
server_key_version: "encryptionkey@0"
I20260929 01:12:09.939139 24139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:09.972350 24139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:09.976984 24139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:09.977293 24139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:09.996340 24139 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:33345
I20260929 01:12:09.996371 24205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:33345 every 8 connection(s)
I20260929 01:12:09.997824 24139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:10.001216 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 24139
I20260929 01:12:10.001433 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:10.001914 24206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:10.002157 5915 external_mini_cluster.cc:1428] Setting key 897c555820d45cc0685891a6f3ebb68c
I20260929 01:12:10.017707 24206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43: Bootstrap starting.
Sep 29 01:12:10 dist-test-slave-41cl krb5kdc[24126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:12:10.020992 24206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:10.022015 24206 log.cc:824] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:10.024961 24206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43: No bootstrap required, opened a new log
I20260929 01:12:10.027020 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:10.004030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.5.198.254:33345)
0929 01:12:10.004898 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:10.004909 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:10.004964 (+ 55us) server_negotiation.cc:407] Connection header received
0929 01:12:10.007379 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:10.007411 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:10.007823 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:10.008177 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:10.010057 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:10.011461 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:10.013441 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:10.014174 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:10.014329 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:10.019410 (+ 5081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:10.019418 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:10.019431 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:10.019475 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:10.023001 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:10.023747 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:10.023763 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:10.023769 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:10.023922 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:10.024311 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:10.024316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:10.024321 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:10.024874 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:10.025186 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:10.025618 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 01:12:10.026043 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":411,"threads_started":1}
I20260929 01:12:10.032801 24206 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4726d0d69274fa2bcf7f0739d9c2b43" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33345 } }
I20260929 01:12:10.033226 24206 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:10.033331 24206 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4726d0d69274fa2bcf7f0739d9c2b43, State: Initialized, Role: FOLLOWER
I20260929 01:12:10.033878 24206 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4726d0d69274fa2bcf7f0739d9c2b43" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33345 } }
I20260929 01:12:10.034124 24206 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:10.034264 24206 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:10.034435 24206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:10.036926 24206 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4726d0d69274fa2bcf7f0739d9c2b43" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33345 } }
I20260929 01:12:10.037663 24206 leader_election.cc:302] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4726d0d69274fa2bcf7f0739d9c2b43; no voters:
I20260929 01:12:10.038373 24206 leader_election.cc:288] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:10.038520 24211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:10.039259 24211 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [term 1 LEADER]: Becoming Leader. State: Replica: a4726d0d69274fa2bcf7f0739d9c2b43, State: Running, Role: LEADER
I20260929 01:12:10.039819 24211 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4726d0d69274fa2bcf7f0739d9c2b43" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33345 } }
I20260929 01:12:10.040740 24206 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:10.043075 24213 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4726d0d69274fa2bcf7f0739d9c2b43. Latest consensus state: current_term: 1 leader_uuid: "a4726d0d69274fa2bcf7f0739d9c2b43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4726d0d69274fa2bcf7f0739d9c2b43" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33345 } } }
I20260929 01:12:10.043490 24213 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:10.047001 24212 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4726d0d69274fa2bcf7f0739d9c2b43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4726d0d69274fa2bcf7f0739d9c2b43" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33345 } } }
I20260929 01:12:10.048146 24212 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:10.048951 24216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:10.054381 24216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:10.064530 24216 catalog_manager.cc:1347] Generated new cluster ID: 1236502468504f93bafa8ca92df6e6de
I20260929 01:12:10.064677 24216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:10.110931 24216 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:10.112838 24216 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:10.128314 24216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43: Generated new TSK 0
I20260929 01:12:10.129413 24216 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:12:10.201309 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33345
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:10.536966 24232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:10.537232 24232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:10.537382 24232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:10.545650 24232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:10.545789 24232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:10.545957 24232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:12:10.557371 24232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:10.558707 24232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:10.560039 24232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:10.573014 24238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:10.573124 24240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:10.573441 24237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:10.574319 24232 server_base.cc:1034] running on GCE node
I20260929 01:12:10.574906 24232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:10.575793 24232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:10.577011 24232 hybrid_clock.cc:648] HybridClock initialized: now 1790644330576982 us; error 53 us; skew 500 ppm
I20260929 01:12:10.577342 24232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:10 dist-test-slave-41cl krb5kdc[24126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:10.580545 24232 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:12:10.581923 24232 webserver.cc:460] Webserver started at http://127.5.198.193:39065/ using document root <none> and password file <none>
I20260929 01:12:10.582588 24232 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:10.582732 24232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:10.583045 24232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:10.585868 24232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "668dc7af663a461a8c8c57766108dd37"
format_stamp: "Formatted at 2026-09-29 01:12:10 on dist-test-slave-41cl"
server_key: "c3eef26b4ba4d452722ef0a2cc324e02"
server_key_iv: "74a70c4fa4d7312f62db29f0c73fd1a3"
server_key_version: "encryptionkey@0"
I20260929 01:12:10.586642 24232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "668dc7af663a461a8c8c57766108dd37"
format_stamp: "Formatted at 2026-09-29 01:12:10 on dist-test-slave-41cl"
server_key: "c3eef26b4ba4d452722ef0a2cc324e02"
server_key_iv: "74a70c4fa4d7312f62db29f0c73fd1a3"
server_key_version: "encryptionkey@0"
I20260929 01:12:10.591678 24232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 01:12:10.595271 24247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:10.596637 24232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:12:10.596825 24232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "668dc7af663a461a8c8c57766108dd37"
format_stamp: "Formatted at 2026-09-29 01:12:10 on dist-test-slave-41cl"
server_key: "c3eef26b4ba4d452722ef0a2cc324e02"
server_key_iv: "74a70c4fa4d7312f62db29f0c73fd1a3"
server_key_version: "encryptionkey@0"
I20260929 01:12:10.597031 24232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:10.623286 24232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:10.628784 24232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:10.629108 24232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:10.630115 24232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:10.631544 24232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:10.631649 24232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:10.631801 24232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:10.631873 24232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:10.667167 24232 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:40549
I20260929 01:12:10.667253 24359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:40549 every 8 connection(s)
I20260929 01:12:10.668504 24232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:12:10.668834 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 24232
I20260929 01:12:10.669023 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:12:10.669607 5915 external_mini_cluster.cc:1428] Setting key e9c4d841618efe785804da88e6186428
Sep 29 01:12:10 dist-test-slave-41cl krb5kdc[24126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:12:10.693580 24362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:10.671447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:47825 (local address 127.5.198.254:33345)
0929 01:12:10.671939 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:10.671946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:10.673066 (+ 1120us) server_negotiation.cc:407] Connection header received
0929 01:12:10.675026 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:10.675034 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:10.675138 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:10.675275 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:10.677363 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:10.678556 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:10.680289 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:10.681126 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:10.681274 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:10.685683 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:10.685693 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:10.685701 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:10.685752 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:10.689588 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:10.690674 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:10.690683 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:10.690689 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:10.690840 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:10.691326 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:10.691334 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:10.691340 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:10.691762 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:10.691923 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:10.692887 (+ 964us) server_negotiation.cc:299] Negotiation successful
0929 01:12:10.693166 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":241,"threads_started":1}
I20260929 01:12:10.694720 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:10.672018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33345 (local address 127.5.198.193:47825)
0929 01:12:10.672889 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:10.672941 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:10.674726 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:10.675743 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:10.675752 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:10.676345 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:10.677143 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:10.677164 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:10.678750 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:10.678753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:10.680126 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:10.680139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:10.681383 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:10.681388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:10.681655 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:10.682505 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:12:10.682529 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:10.685365 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:10.689965 (+ 4600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:10.689973 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:10.689992 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:10.690520 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:10.690957 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:10.690961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:10.690963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:10.691193 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:10.691902 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:10.691908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:10.692275 (+ 367us) client_negotiation.cc:769] Sending connection context
0929 01:12:10.692748 (+ 473us) client_negotiation.cc:240] Negotiation successful
0929 01:12:10.693406 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":392,"threads_started":1}
I20260929 01:12:10.696523 24360 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33345
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
I20260929 01:12:10.696924 24360 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:10.697690 24360 heartbeater.cc:511] Master 127.5.198.254:33345 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
I20260929 01:12:10.700424 24171 ts_manager.cc:194] Registered new tserver with Master: 668dc7af663a461a8c8c57766108dd37 (127.5.198.193:40549)
I20260929 01:12:10.702123 24171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:47825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:12:10.733021 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33345
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:11.071573 24366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:11.071827 24366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:11.071918 24366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:11.079514 24366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:11.079617 24366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:11.079722 24366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:12:11.089124 24366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:11.090266 24366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:11.091450 24366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:11.104202 24371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:11.104269 24374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:11.104244 24372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:11.105121 24366 server_base.cc:1034] running on GCE node
I20260929 01:12:11.105614 24366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:11.106487 24366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:11.107698 24366 hybrid_clock.cc:648] HybridClock initialized: now 1790644331107656 us; error 35 us; skew 500 ppm
I20260929 01:12:11.108058 24366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:11 dist-test-slave-41cl krb5kdc[24126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:11.115654 24366 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:12:11.117298 24366 webserver.cc:460] Webserver started at http://127.5.198.194:42303/ using document root <none> and password file <none>
I20260929 01:12:11.117976 24366 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:11.118093 24366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:11.118373 24366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:11.121421 24366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "235be439bc6e4ebdbfd9e0591626dff4"
format_stamp: "Formatted at 2026-09-29 01:12:11 on dist-test-slave-41cl"
server_key: "9d71712142251acfbac3f3f8695c1275"
server_key_iv: "8bd4b48fc573bcf44b5b1d8978b3991d"
server_key_version: "encryptionkey@0"
I20260929 01:12:11.122428 24366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "235be439bc6e4ebdbfd9e0591626dff4"
format_stamp: "Formatted at 2026-09-29 01:12:11 on dist-test-slave-41cl"
server_key: "9d71712142251acfbac3f3f8695c1275"
server_key_iv: "8bd4b48fc573bcf44b5b1d8978b3991d"
server_key_version: "encryptionkey@0"
I20260929 01:12:11.128115 24366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:12:11.131783 24381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:11.133229 24366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:12:11.133409 24366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "235be439bc6e4ebdbfd9e0591626dff4"
format_stamp: "Formatted at 2026-09-29 01:12:11 on dist-test-slave-41cl"
server_key: "9d71712142251acfbac3f3f8695c1275"
server_key_iv: "8bd4b48fc573bcf44b5b1d8978b3991d"
server_key_version: "encryptionkey@0"
I20260929 01:12:11.133601 24366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:11.153751 24366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:11.157759 24366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:11.158046 24366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:11.159018 24366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:11.160291 24366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:11.160362 24366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:11.160562 24366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:11.160638 24366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:11.194214 24366 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:40835
I20260929 01:12:11.194249 24493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:40835 every 8 connection(s)
I20260929 01:12:11.195485 24366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:12:11.196588 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 24366
I20260929 01:12:11.196775 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:12:11.197275 5915 external_mini_cluster.cc:1428] Setting key b75b5b0b680f30e590e9d9d24376385f
Sep 29 01:12:11 dist-test-slave-41cl krb5kdc[24126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:12:11.218377 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:11.198462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:56113 (local address 127.5.198.254:33345)
0929 01:12:11.198942 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:11.198948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:11.199972 (+ 1024us) server_negotiation.cc:407] Connection header received
0929 01:12:11.202104 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:11.202110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:11.202214 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:11.202335 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:11.203787 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.204570 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.206204 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.206910 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.207061 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:11.210950 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:11.210960 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:11.210968 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:11.211016 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:11.214919 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:11.215680 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:11.215688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:11.215695 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:11.215851 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:11.216296 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:11.216304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:11.216310 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:11.216752 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:11.216917 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:11.217621 (+ 704us) server_negotiation.cc:299] Negotiation successful
0929 01:12:11.217875 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
I20260929 01:12:11.219015 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:11.198935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33345 (local address 127.5.198.194:56113)
0929 01:12:11.199833 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:11.199870 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:11.201847 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:11.202570 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:11.202579 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:11.203076 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:11.203607 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:11.203624 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.204717 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.204722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:11.206058 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:11.206070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.207171 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.207176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:11.207387 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:11.208096 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:12:11.208123 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:11.210682 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:11.215077 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:11.215092 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:11.215106 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:11.215551 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:11.215961 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:11.215964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:11.215966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:11.216180 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:11.216883 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:11.216889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:11.217250 (+ 361us) client_negotiation.cc:769] Sending connection context
0929 01:12:11.217482 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 01:12:11.217876 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":361,"threads_started":1}
I20260929 01:12:11.220753 24494 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33345
I20260929 01:12:11.221061 24494 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:11.221838 24494 heartbeater.cc:511] Master 127.5.198.254:33345 requested a full tablet report, sending...
I20260929 01:12:11.223374 24171 ts_manager.cc:194] Registered new tserver with Master: 235be439bc6e4ebdbfd9e0591626dff4 (127.5.198.194:40835)
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
I20260929 01:12:11.224309 24171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:56113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:12:11.256220 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33345
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:11.584764 24500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:11.585034 24500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:11.585139 24500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:11.592180 24500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:11.592283 24500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:11.592394 24500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:12:11.602504 24500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:33345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:11.603773 24500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:11.605041 24500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:11.619942 24505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:11.620011 24506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:11.620011 24508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:11.621528 24500 server_base.cc:1034] running on GCE node
I20260929 01:12:11.622267 24500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:11.623351 24500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:11.624714 24500 hybrid_clock.cc:648] HybridClock initialized: now 1790644331624667 us; error 33 us; skew 500 ppm
I20260929 01:12:11.625128 24500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:11 dist-test-slave-41cl krb5kdc[24126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:11.628698 24500 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:12:11.630541 24500 webserver.cc:460] Webserver started at http://127.5.198.195:38939/ using document root <none> and password file <none>
I20260929 01:12:11.631291 24500 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:11.631456 24500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:11.631794 24500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:11.634847 24500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "0e77817fa4f544bda77cff00a65fae1d"
format_stamp: "Formatted at 2026-09-29 01:12:11 on dist-test-slave-41cl"
server_key: "7f0b64aede57c1b9472ed3cc5dbaa1e4"
server_key_iv: "1477bc545160d7dc5cc4e86cebec69b9"
server_key_version: "encryptionkey@0"
I20260929 01:12:11.635653 24500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "0e77817fa4f544bda77cff00a65fae1d"
format_stamp: "Formatted at 2026-09-29 01:12:11 on dist-test-slave-41cl"
server_key: "7f0b64aede57c1b9472ed3cc5dbaa1e4"
server_key_iv: "1477bc545160d7dc5cc4e86cebec69b9"
server_key_version: "encryptionkey@0"
I20260929 01:12:11.641710 24500 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:12:11.645913 24515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:11.647605 24500 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260929 01:12:11.647816 24500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "0e77817fa4f544bda77cff00a65fae1d"
format_stamp: "Formatted at 2026-09-29 01:12:11 on dist-test-slave-41cl"
server_key: "7f0b64aede57c1b9472ed3cc5dbaa1e4"
server_key_iv: "1477bc545160d7dc5cc4e86cebec69b9"
server_key_version: "encryptionkey@0"
I20260929 01:12:11.648059 24500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:11.670344 24500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:11.674742 24500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:11.675036 24500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:11.675985 24500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:11.677381 24500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:11.677488 24500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:11.677620 24500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:11.677696 24500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:11.705230 24360 heartbeater.cc:503] Master 127.5.198.254:33345 was elected leader, sending a full tablet report...
I20260929 01:12:11.712090 24500 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:39155
I20260929 01:12:11.712170 24627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:39155 every 8 connection(s)
I20260929 01:12:11.713450 24500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:12:11.721694 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 24500
I20260929 01:12:11.721865 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:12:11.722368 5915 external_mini_cluster.cc:1428] Setting key 55214e84f47deb936d04f9e677908bce
Sep 29 01:12:11 dist-test-slave-41cl krb5kdc[24126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:12:11.733335 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:11.716242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:48707 (local address 127.5.198.254:33345)
0929 01:12:11.716451 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:11.716458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:11.717715 (+ 1257us) server_negotiation.cc:407] Connection header received
0929 01:12:11.719513 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:11.719520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:11.719616 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:11.719741 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:11.721315 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.722183 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.723937 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.724339 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.724470 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:11.727902 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:11.727907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:11.727910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:11.727943 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:11.730404 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:11.731108 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:11.731120 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:11.731123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:11.731232 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:11.731647 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:11.731651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:11.731657 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:11.731900 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:11.732071 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:11.732802 (+ 731us) server_negotiation.cc:299] Negotiation successful
0929 01:12:11.733018 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 01:12:11.734194 24630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:11.716833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:33345 (local address 127.5.198.195:48707)
0929 01:12:11.717553 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:11.717594 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:11.719246 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:11.719962 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:11.719971 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:11.720542 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:11.721159 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:11.721179 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.722338 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.722343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:11.723799 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:11.723812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.724577 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.724585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:11.724806 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:11.725476 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:12:11.725497 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:11.727694 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:11.730545 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:11.730555 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:11.730570 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:11.730978 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:11.731340 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:11.731346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:11.731349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:11.731529 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:11.732026 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:11.732034 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:11.732350 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 01:12:11.732678 (+ 328us) client_negotiation.cc:240] Negotiation successful
0929 01:12:11.733173 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":288,"threads_started":1}
I20260929 01:12:11.735608 24628 heartbeater.cc:347] Connected to a master server at 127.5.198.254:33345
I20260929 01:12:11.735944 24628 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:11.736756 24628 heartbeater.cc:511] Master 127.5.198.254:33345 requested a full tablet report, sending...
I20260929 01:12:11.738214 24171 ts_manager.cc:194] Registered new tserver with Master: 0e77817fa4f544bda77cff00a65fae1d (127.5.198.195:39155)
I20260929 01:12:11.739032 24171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:48707
I20260929 01:12:11.749490 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:12:11.767099 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:11.753684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.5.198.254:33345)
0929 01:12:11.753879 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:11.753888 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:11.754214 (+ 326us) server_negotiation.cc:407] Connection header received
0929 01:12:11.754402 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:11.754407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:11.754487 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:11.754599 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:11.756007 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.756829 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.758180 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.758780 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.758911 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:11.761803 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:11.761809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:11.761814 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:11.761856 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:11.764278 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:11.765044 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:11.765049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:11.765053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:11.765188 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:11.765642 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:11.765653 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:11.765656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:11.765924 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:11.766202 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:11.766610 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 01:12:11.766796 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:12:11.770996 24171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:12:11.777844 24171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:12:11.802441 24641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:11.789407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:39155 (local address 127.0.0.1:46334)
0929 01:12:11.792810 (+ 3403us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:11.792848 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:11.793068 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:11.794138 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:11.794146 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:11.794190 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:11.794712 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:11.794729 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.796523 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.796529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:11.799329 (+ 2800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:11.799352 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.801458 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.801463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:11.801693 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:11.801761 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 01:12:11.801924 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 01:12:11.802079 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3262,"mutex_wait_us":31,"spinlock_wait_cycles":896,"thread_start_us":3099,"threads_started":1}
I20260929 01:12:11.803071 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:11.789441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37694 (local address 127.5.198.194:40835)
0929 01:12:11.792134 (+ 2693us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:11.792145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:11.793388 (+ 1243us) server_negotiation.cc:407] Connection header received
0929 01:12:11.793493 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:11.793500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:11.793701 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:11.793872 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:11.795524 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.797252 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.799927 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.801584 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.801733 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:11.801949 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:11.802584 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 01:12:11.802749 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2506,"thread_start_us":2383,"threads_started":1}
I20260929 01:12:11.803372 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:11.793535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44324 (local address 127.5.198.193:40549)
0929 01:12:11.794650 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:11.794656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:11.795386 (+ 730us) server_negotiation.cc:407] Connection header received
0929 01:12:11.795600 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:11.795610 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:11.795836 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:11.796067 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:11.796625 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.798161 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.800159 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.801750 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.801882 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:11.802103 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:11.802851 (+ 748us) server_negotiation.cc:299] Negotiation successful
0929 01:12:11.803024 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":955,"thread_start_us":820,"threads_started":1}
I20260929 01:12:11.803465 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:11.789790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.5.198.195:39155)
0929 01:12:11.791663 (+ 1873us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:11.791677 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:11.792943 (+ 1266us) server_negotiation.cc:407] Connection header received
0929 01:12:11.793317 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:11.793327 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:11.793603 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:11.793807 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:11.794941 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.796355 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.799529 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.800966 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.801087 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:11.801327 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:11.802869 (+ 1542us) server_negotiation.cc:299] Negotiation successful
0929 01:12:11.803020 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1609,"thread_start_us":1403,"threads_started":1}
I20260929 01:12:11.803100 24640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:11.789265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:40835 (local address 127.0.0.1:37694)
0929 01:12:11.792810 (+ 3545us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:11.792849 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:11.793036 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:11.794138 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:11.794146 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:11.794197 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:11.795302 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:11.795313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.797430 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.797435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:11.799771 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:11.799782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.802123 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.802132 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:11.802361 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:11.802404 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 01:12:11.802511 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 01:12:11.802661 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3259,"mutex_wait_us":611,"spinlock_wait_cycles":6144,"thread_start_us":2937,"threads_started":1}
I20260929 01:12:11.803982 24645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:11.794615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:40549 (local address 127.0.0.1:44324)
0929 01:12:11.795249 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:11.795269 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:11.795397 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:11.796061 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:11.796065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:11.796089 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:11.796387 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:11.796401 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.798303 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.798309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:11.799952 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:11.799968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.802256 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.802262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:11.802479 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:11.802520 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 01:12:11.802597 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 01:12:11.802702 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":294,"threads_started":1}
I20260929 01:12:11.807530 24563 tablet_service.cc:1467] Processing CreateTablet for tablet 599e30a7d725413eacf11ef1ec9f0bf0 (DEFAULT_TABLE table=test-table [id=906f3af761eb4d6db8abf6f1432a78d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:11.807550 24429 tablet_service.cc:1467] Processing CreateTablet for tablet 599e30a7d725413eacf11ef1ec9f0bf0 (DEFAULT_TABLE table=test-table [id=906f3af761eb4d6db8abf6f1432a78d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:11.807541 24295 tablet_service.cc:1467] Processing CreateTablet for tablet 599e30a7d725413eacf11ef1ec9f0bf0 (DEFAULT_TABLE table=test-table [id=906f3af761eb4d6db8abf6f1432a78d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:11.808902 24563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599e30a7d725413eacf11ef1ec9f0bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:11.808936 24429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599e30a7d725413eacf11ef1ec9f0bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:11.808936 24295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599e30a7d725413eacf11ef1ec9f0bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:11.821888 24646 tablet_bootstrap.cc:492] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d: Bootstrap starting.
I20260929 01:12:11.825241 24646 tablet_bootstrap.cc:654] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:11.826305 24646 log.cc:824] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:11.827322 24648 tablet_bootstrap.cc:492] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4: Bootstrap starting.
I20260929 01:12:11.828311 24647 tablet_bootstrap.cc:492] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37: Bootstrap starting.
I20260929 01:12:11.831218 24648 tablet_bootstrap.cc:654] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:11.832176 24648 log.cc:824] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:11.832886 24647 tablet_bootstrap.cc:654] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:11.834164 24647 log.cc:824] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:11.835386 24648 tablet_bootstrap.cc:492] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4: No bootstrap required, opened a new log
I20260929 01:12:11.835443 24646 tablet_bootstrap.cc:492] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d: No bootstrap required, opened a new log
I20260929 01:12:11.835693 24646 ts_tablet_manager.cc:1397] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20260929 01:12:11.835723 24648 ts_tablet_manager.cc:1397] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20260929 01:12:11.837361 24647 tablet_bootstrap.cc:492] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37: No bootstrap required, opened a new log
I20260929 01:12:11.837633 24647 ts_tablet_manager.cc:1397] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260929 01:12:11.844106 24646 raft_consensus.cc:348] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235be439bc6e4ebdbfd9e0591626dff4" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40835 } } peers { permanent_uuid: "0e77817fa4f544bda77cff00a65fae1d" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39155 } } peers { permanent_uuid: "668dc7af663a461a8c8c57766108dd37" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 40549 } }
I20260929 01:12:11.844501 24646 raft_consensus.cc:374] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:11.844599 24646 raft_consensus.cc:729] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e77817fa4f544bda77cff00a65fae1d, State: Initialized, Role: FOLLOWER
I20260929 01:12:11.845216 24646 consensus_queue.cc:260] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235be439bc6e4ebdbfd9e0591626dff4" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40835 } } peers { permanent_uuid: "0e77817fa4f544bda77cff00a65fae1d" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39155 } } peers { permanent_uuid: "668dc7af663a461a8c8c57766108dd37" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 40549 } }
I20260929 01:12:11.846269 24628 heartbeater.cc:503] Master 127.5.198.254:33345 was elected leader, sending a full tablet report...
I20260929 01:12:11.846961 24646 ts_tablet_manager.cc:1428] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 01:12:11.848129 24648 raft_consensus.cc:348] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235be439bc6e4ebdbfd9e0591626dff4" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40835 } } peers { permanent_uuid: "0e77817fa4f544bda77cff00a65fae1d" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39155 } } peers { permanent_uuid: "668dc7af663a461a8c8c57766108dd37" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 40549 } }
I20260929 01:12:11.848677 24648 raft_consensus.cc:374] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:11.848809 24648 raft_consensus.cc:729] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 235be439bc6e4ebdbfd9e0591626dff4, State: Initialized, Role: FOLLOWER
I20260929 01:12:11.849261 24647 raft_consensus.cc:348] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235be439bc6e4ebdbfd9e0591626dff4" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40835 } } peers { permanent_uuid: "0e77817fa4f544bda77cff00a65fae1d" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39155 } } peers { permanent_uuid: "668dc7af663a461a8c8c57766108dd37" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 40549 } }
I20260929 01:12:11.849442 24648 consensus_queue.cc:260] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235be439bc6e4ebdbfd9e0591626dff4" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40835 } } peers { permanent_uuid: "0e77817fa4f544bda77cff00a65fae1d" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39155 } } peers { permanent_uuid: "668dc7af663a461a8c8c57766108dd37" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 40549 } }
I20260929 01:12:11.849720 24647 raft_consensus.cc:374] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:11.849828 24647 raft_consensus.cc:729] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668dc7af663a461a8c8c57766108dd37, State: Initialized, Role: FOLLOWER
I20260929 01:12:11.850835 24647 consensus_queue.cc:260] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235be439bc6e4ebdbfd9e0591626dff4" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40835 } } peers { permanent_uuid: "0e77817fa4f544bda77cff00a65fae1d" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39155 } } peers { permanent_uuid: "668dc7af663a461a8c8c57766108dd37" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 40549 } }
I20260929 01:12:11.853200 24647 ts_tablet_manager.cc:1428] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37: Time spent starting tablet: real 0.015s user 0.004s sys 0.012s
I20260929 01:12:11.854215 24494 heartbeater.cc:503] Master 127.5.198.254:33345 was elected leader, sending a full tablet report...
I20260929 01:12:11.854878 24648 ts_tablet_manager.cc:1428] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4: Time spent starting tablet: real 0.019s user 0.015s sys 0.004s
I20260929 01:12:11.910220 24654 raft_consensus.cc:482] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:12:11.910506 24654 raft_consensus.cc:504] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235be439bc6e4ebdbfd9e0591626dff4" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40835 } } peers { permanent_uuid: "0e77817fa4f544bda77cff00a65fae1d" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39155 } } peers { permanent_uuid: "668dc7af663a461a8c8c57766108dd37" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 40549 } }
I20260929 01:12:11.912333 24654 leader_election.cc:288] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 235be439bc6e4ebdbfd9e0591626dff4 (127.5.198.194:40835), 0e77817fa4f544bda77cff00a65fae1d (127.5.198.195:39155)
W20260929 01:12:11.921216 24361 tablet.cc:2369] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:12:11.922016 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:11.912766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:39219 (local address 127.5.198.194:40835)
0929 01:12:11.913015 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:11.913022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:11.913656 (+ 634us) server_negotiation.cc:407] Connection header received
0929 01:12:11.913989 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:11.913995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:11.914087 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:11.914217 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:11.914956 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.916107 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.919400 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.920736 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.920931 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:11.921008 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:11.921612 (+ 604us) server_negotiation.cc:299] Negotiation successful
0929 01:12:11.921748 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 01:12:11.922142 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:11.912926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:60469 (local address 127.5.198.195:39155)
0929 01:12:11.913239 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:11.913249 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:11.913947 (+ 698us) server_negotiation.cc:407] Connection header received
0929 01:12:11.914156 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:11.914161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:11.914234 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:11.914321 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:11.915157 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.916229 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.919400 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.920784 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.920937 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:11.921008 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:11.921632 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 01:12:11.921787 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260929 01:12:11.922159 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:11.912575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:40835 (local address 127.5.198.193:39219)
0929 01:12:11.913531 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:11.913555 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:11.913794 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:11.914228 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:11.914233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:11.914259 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:11.914791 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:11.914809 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.916258 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.916262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:11.919188 (+ 2926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:11.919205 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.921200 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.921205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:11.921436 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:11.921473 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 01:12:11.921582 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 01:12:11.921722 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":248,"thread_start_us":630,"threads_started":1}
I20260929 01:12:11.922778 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:11.912931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:39155 (local address 127.5.198.193:60469)
0929 01:12:11.913828 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:11.913843 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:11.914001 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:11.914434 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:11.914448 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:11.914475 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:11.915010 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:11.915020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.916362 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.916368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:11.919188 (+ 2820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:11.919205 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.921051 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.921059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:11.921301 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:11.921404 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 01:12:11.922276 (+ 872us) client_negotiation.cc:240] Negotiation successful
0929 01:12:11.922407 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":176,"thread_start_us":535,"threads_started":1}
I20260929 01:12:11.923569 24449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599e30a7d725413eacf11ef1ec9f0bf0" candidate_uuid: "668dc7af663a461a8c8c57766108dd37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "235be439bc6e4ebdbfd9e0591626dff4" is_pre_election: true
I20260929 01:12:11.923767 24583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599e30a7d725413eacf11ef1ec9f0bf0" candidate_uuid: "668dc7af663a461a8c8c57766108dd37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e77817fa4f544bda77cff00a65fae1d" is_pre_election: true
I20260929 01:12:11.924230 24449 raft_consensus.cc:2454] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668dc7af663a461a8c8c57766108dd37 in term 0.
I20260929 01:12:11.924319 24583 raft_consensus.cc:2454] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668dc7af663a461a8c8c57766108dd37 in term 0.
I20260929 01:12:11.925084 24248 leader_election.cc:302] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 235be439bc6e4ebdbfd9e0591626dff4, 668dc7af663a461a8c8c57766108dd37; no voters:
I20260929 01:12:11.925531 24654 raft_consensus.cc:2781] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:12:11.925683 24654 raft_consensus.cc:482] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:12:11.925808 24654 raft_consensus.cc:3037] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:11.928089 24654 raft_consensus.cc:504] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235be439bc6e4ebdbfd9e0591626dff4" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40835 } } peers { permanent_uuid: "0e77817fa4f544bda77cff00a65fae1d" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39155 } } peers { permanent_uuid: "668dc7af663a461a8c8c57766108dd37" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 40549 } }
I20260929 01:12:11.928771 24654 leader_election.cc:288] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [CANDIDATE]: Term 1 election: Requested vote from peers 235be439bc6e4ebdbfd9e0591626dff4 (127.5.198.194:40835), 0e77817fa4f544bda77cff00a65fae1d (127.5.198.195:39155)
I20260929 01:12:11.929307 24449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599e30a7d725413eacf11ef1ec9f0bf0" candidate_uuid: "668dc7af663a461a8c8c57766108dd37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "235be439bc6e4ebdbfd9e0591626dff4"
I20260929 01:12:11.929345 24583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599e30a7d725413eacf11ef1ec9f0bf0" candidate_uuid: "668dc7af663a461a8c8c57766108dd37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e77817fa4f544bda77cff00a65fae1d"
I20260929 01:12:11.929523 24449 raft_consensus.cc:3037] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:11.929567 24583 raft_consensus.cc:3037] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:11.931756 24449 raft_consensus.cc:2454] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668dc7af663a461a8c8c57766108dd37 in term 1.
I20260929 01:12:11.932065 24583 raft_consensus.cc:2454] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668dc7af663a461a8c8c57766108dd37 in term 1.
I20260929 01:12:11.932228 24248 leader_election.cc:302] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 235be439bc6e4ebdbfd9e0591626dff4, 668dc7af663a461a8c8c57766108dd37; no voters:
I20260929 01:12:11.932582 24654 raft_consensus.cc:2781] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:11.933089 24654 raft_consensus.cc:686] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [term 1 LEADER]: Becoming Leader. State: Replica: 668dc7af663a461a8c8c57766108dd37, State: Running, Role: LEADER
I20260929 01:12:11.933549 24654 consensus_queue.cc:237] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235be439bc6e4ebdbfd9e0591626dff4" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40835 } } peers { permanent_uuid: "0e77817fa4f544bda77cff00a65fae1d" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39155 } } peers { permanent_uuid: "668dc7af663a461a8c8c57766108dd37" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 40549 } }
I20260929 01:12:11.938483 24170 catalog_manager.cc:5495] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668dc7af663a461a8c8c57766108dd37 (127.5.198.193). New cstate: current_term: 1 leader_uuid: "668dc7af663a461a8c8c57766108dd37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "235be439bc6e4ebdbfd9e0591626dff4" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e77817fa4f544bda77cff00a65fae1d" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668dc7af663a461a8c8c57766108dd37" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 40549 } health_report { overall_health: HEALTHY } } }
W20260929 01:12:11.947763 24495 tablet.cc:2369] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:12:11.957139 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:11.951666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44334 (local address 127.5.198.193:40549)
0929 01:12:11.951873 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:11.951881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:11.951906 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:12:11.952209 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:11.952214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:11.952286 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:11.952431 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:12:11.952885 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.953589 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.954938 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:11.955327 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:11.955469 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:11.956035 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:12:11.956562 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:12:11.956755 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:11.956865 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 01:12:11.956910 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:12:11.965054 24583 raft_consensus.cc:1261] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d [term 1 FOLLOWER]: Refusing update from remote peer 668dc7af663a461a8c8c57766108dd37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:12:11.965135 24449 raft_consensus.cc:1261] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4 [term 1 FOLLOWER]: Refusing update from remote peer 668dc7af663a461a8c8c57766108dd37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 01:12:11.965032 24629 tablet.cc:2369] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:12:11.966046 24657 consensus_queue.cc:1035] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "235be439bc6e4ebdbfd9e0591626dff4" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 40835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:12:11.966413 24654 consensus_queue.cc:1035] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e77817fa4f544bda77cff00a65fae1d" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:12:11.993608 24171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:12:11.993908 24171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:12:11.997989 24171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4726d0d69274fa2bcf7f0739d9c2b43: Sending DeleteTablet for 3 replicas of tablet 599e30a7d725413eacf11ef1ec9f0bf0
I20260929 01:12:11.999105 24563 tablet_service.cc:1514] Processing DeleteTablet for tablet 599e30a7d725413eacf11ef1ec9f0bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:11 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:46334
I20260929 01:12:11.999852 24429 tablet_service.cc:1514] Processing DeleteTablet for tablet 599e30a7d725413eacf11ef1ec9f0bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:11 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:37694
I20260929 01:12:12.000084 24670 tablet_replica.cc:330] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d: stopping tablet replica
I20260929 01:12:12.000587 24670 raft_consensus.cc:2229] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:12:12.000967 24295 tablet_service.cc:1514] Processing DeleteTablet for tablet 599e30a7d725413eacf11ef1ec9f0bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:11 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:44324
I20260929 01:12:12.001163 24670 raft_consensus.cc:2258] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:12.001915 24671 tablet_replica.cc:330] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4: stopping tablet replica
I20260929 01:12:12.002271 24671 raft_consensus.cc:2229] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:12:12.002707 24671 raft_consensus.cc:2258] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:12.003726 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 24232
I20260929 01:12:12.010432 24670 ts_tablet_manager.cc:1905] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:12.010491 24671 ts_tablet_manager.cc:1905] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:12.023361 24672 tablet_replica.cc:330] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37: stopping tablet replica
I20260929 01:12:12.023973 24672 raft_consensus.cc:2229] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:12:12.024821 24672 raft_consensus.cc:2258] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:12.027004 24672 ts_tablet_manager.cc:1905] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:12.027781 24671 ts_tablet_manager.cc:1918] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:12:12.027999 24671 log.cc:1196] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/599e30a7d725413eacf11ef1ec9f0bf0
I20260929 01:12:12.028684 24671 ts_tablet_manager.cc:1939] T 599e30a7d725413eacf11ef1ec9f0bf0 P 235be439bc6e4ebdbfd9e0591626dff4: Deleting consensus metadata
I20260929 01:12:12.030618 24155 catalog_manager.cc:4841] TS 235be439bc6e4ebdbfd9e0591626dff4 (127.5.198.194:40835): tablet 599e30a7d725413eacf11ef1ec9f0bf0 (table test-table [id=906f3af761eb4d6db8abf6f1432a78d5]) successfully deleted
I20260929 01:12:12.032143 24670 ts_tablet_manager.cc:1918] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:12:12.032325 24670 log.cc:1196] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/599e30a7d725413eacf11ef1ec9f0bf0
I20260929 01:12:12.034596 24672 ts_tablet_manager.cc:1918] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:12:12.034757 24672 log.cc:1196] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/599e30a7d725413eacf11ef1ec9f0bf0
I20260929 01:12:12.034749 24670 ts_tablet_manager.cc:1939] T 599e30a7d725413eacf11ef1ec9f0bf0 P 0e77817fa4f544bda77cff00a65fae1d: Deleting consensus metadata
I20260929 01:12:12.035234 24672 ts_tablet_manager.cc:1939] T 599e30a7d725413eacf11ef1ec9f0bf0 P 668dc7af663a461a8c8c57766108dd37: Deleting consensus metadata
I20260929 01:12:12.036705 24157 catalog_manager.cc:4841] TS 0e77817fa4f544bda77cff00a65fae1d (127.5.198.195:39155): tablet 599e30a7d725413eacf11ef1ec9f0bf0 (table test-table [id=906f3af761eb4d6db8abf6f1432a78d5]) successfully deleted
I20260929 01:12:12.036872 24155 catalog_manager.cc:4841] TS 668dc7af663a461a8c8c57766108dd37 (127.5.198.193:40549): tablet 599e30a7d725413eacf11ef1ec9f0bf0 (table test-table [id=906f3af761eb4d6db8abf6f1432a78d5]) successfully deleted
Sep 29 01:12:12 dist-test-slave-41cl krb5kdc[24126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:12:12.048707 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:12.007098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.5.198.193:40549)
0929 01:12:12.007338 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:12.007348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:12.024984 (+ 17636us) server_negotiation.cc:407] Connection header received
0929 01:12:12.025526 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:12.025534 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:12.025642 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:12.025716 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:12.027609 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:12.028854 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:12.032596 (+ 3742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:12.033350 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:12.033459 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:12.041235 (+ 7776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:12.041251 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:12.041274 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:12.041317 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:12.045584 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:12.046340 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:12.046350 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:12.046354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:12.046547 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:12.046899 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:12.046905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:12.046908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:12.047373 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:12.047704 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:12.048064 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 01:12:12.048306 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 01:12:12.049558 24355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:12.148747 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 24232
I20260929 01:12:12.167551 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 24366
Sep 29 01:12:12 dist-test-slave-41cl krb5kdc[24126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:12:12.182955 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:12.168361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.5.198.194:40835)
0929 01:12:12.168583 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:12.168600 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:12.168619 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:12:12.168744 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:12.168748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:12.168814 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:12.168914 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:12.170117 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:12.170856 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:12.172117 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:12.172552 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:12.172655 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:12.177401 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:12.177421 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:12.177448 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:12.177483 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:12.180085 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:12.181031 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:12.181039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:12.181041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:12.181165 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:12.181533 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:12.181537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:12.181539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:12.181902 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:12.182162 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:12.182541 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 01:12:12.182700 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 01:12:12.183609 24489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:12.272342 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 24366
I20260929 01:12:12.290932 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 24500
Sep 29 01:12:12 dist-test-slave-41cl krb5kdc[24126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:12:12.306857 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:12.291796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46340 (local address 127.5.198.195:39155)
0929 01:12:12.291961 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:12.291965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:12.291982 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:12:12.292038 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:12.292041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:12.292100 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:12.292194 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:12.293680 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:12.294430 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:12.295745 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:12.296181 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:12.296278 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:12.301262 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:12.301277 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:12.301295 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:12.301336 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:12.304113 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:12.304957 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:12.304964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:12.304966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:12.305149 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:12.305494 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:12.305498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:12.305501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:12.305832 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:12.306082 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:12.306411 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 01:12:12.306612 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:12:12.307590 24623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:12.395911 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 24500
I20260929 01:12:12.412407 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 24139
I20260929 01:12:12.413650 24201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:12.523926 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 24139
2026-09-29T01:12:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4343 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T01:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:12Z Disabled control of system clock
I20260929 01:12:12.598941 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:42869
--webserver_interface=127.5.198.254
--webserver_port=0
--builtin_ntp_servers=127.5.198.212:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:42869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:12:12.931816 24681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:12.932082 24681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:12.932255 24681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:12.939462 24681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:12.939574 24681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:12.939677 24681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:12.939790 24681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:12.939870 24681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:12.948905 24681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:42869
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:42869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:12.950335 24681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:12.952106 24681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:12.963878 24686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:12.963780 24687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:12.963801 24689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:12.964774 24681 server_base.cc:1034] running on GCE node
I20260929 01:12:12.965561 24681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:12.967031 24681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:12.968277 24681 hybrid_clock.cc:648] HybridClock initialized: now 1790644332968239 us; error 59 us; skew 500 ppm
I20260929 01:12:12.968724 24681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:12:12.971035 24681 webserver.cc:460] Webserver started at http://127.5.198.254:36035/ using document root <none> and password file <none>
I20260929 01:12:12.971745 24681 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:12.971873 24681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:12.972200 24681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:12.975109 24681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "cae2d6cbbc2f466ba99306c47663044e"
format_stamp: "Formatted at 2026-09-29 01:12:12 on dist-test-slave-41cl"
server_key: "ab8690f6a554f37ce1456121e5306c71"
server_key_iv: "877d051af5c4b226e894dde2751efff8"
server_key_version: "encryptionkey@0"
I20260929 01:12:12.975991 24681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "cae2d6cbbc2f466ba99306c47663044e"
format_stamp: "Formatted at 2026-09-29 01:12:12 on dist-test-slave-41cl"
server_key: "ab8690f6a554f37ce1456121e5306c71"
server_key_iv: "877d051af5c4b226e894dde2751efff8"
server_key_version: "encryptionkey@0"
I20260929 01:12:12.981233 24681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:12:12.984653 24695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:12.986032 24681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:12:12.986243 24681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "cae2d6cbbc2f466ba99306c47663044e"
format_stamp: "Formatted at 2026-09-29 01:12:12 on dist-test-slave-41cl"
server_key: "ab8690f6a554f37ce1456121e5306c71"
server_key_iv: "877d051af5c4b226e894dde2751efff8"
server_key_version: "encryptionkey@0"
I20260929 01:12:12.986466 24681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:12.998816 24681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:12.999640 24681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:12.999938 24681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:13.019341 24681 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:42869
I20260929 01:12:13.019362 24746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:42869 every 8 connection(s)
I20260929 01:12:13.020730 24681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:13.021096 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 24681
I20260929 01:12:13.021286 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:13.021827 5915 external_mini_cluster.cc:1428] Setting key 81acbadc8f7ed956cb6f4b0bcf1a465b
I20260929 01:12:13.025167 24747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:13.035420 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:13.023772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.5.198.254:42869)
0929 01:12:13.024781 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:13.024792 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:13.024832 (+ 40us) server_negotiation.cc:407] Connection header received
0929 01:12:13.026538 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:13.026561 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:13.026979 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:13.027296 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:13.027974 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:13.029337 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:13.031283 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:13.031902 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:13.032079 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:13.033280 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:13.033288 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:13.033301 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:13.033814 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:13.033885 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:13.033904 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:13.034006 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:13.034246 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 01:12:13.034491 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":381,"threads_started":1}
I20260929 01:12:13.037484 24747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e: Bootstrap starting.
I20260929 01:12:13.040922 24747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:13.041814 24747 log.cc:824] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:13.044330 24747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e: No bootstrap required, opened a new log
I20260929 01:12:13.051600 24747 raft_consensus.cc:348] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae2d6cbbc2f466ba99306c47663044e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42869 } }
I20260929 01:12:13.051899 24747 raft_consensus.cc:374] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:13.051968 24747 raft_consensus.cc:729] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae2d6cbbc2f466ba99306c47663044e, State: Initialized, Role: FOLLOWER
I20260929 01:12:13.052418 24747 consensus_queue.cc:260] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae2d6cbbc2f466ba99306c47663044e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42869 } }
I20260929 01:12:13.052672 24747 raft_consensus.cc:388] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:13.052762 24747 raft_consensus.cc:482] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:13.052883 24747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:13.054874 24747 raft_consensus.cc:504] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae2d6cbbc2f466ba99306c47663044e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42869 } }
I20260929 01:12:13.055418 24747 leader_election.cc:302] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cae2d6cbbc2f466ba99306c47663044e; no voters:
I20260929 01:12:13.056084 24747 leader_election.cc:288] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:13.056257 24752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:13.057106 24752 raft_consensus.cc:686] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [term 1 LEADER]: Becoming Leader. State: Replica: cae2d6cbbc2f466ba99306c47663044e, State: Running, Role: LEADER
I20260929 01:12:13.057641 24752 consensus_queue.cc:237] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae2d6cbbc2f466ba99306c47663044e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42869 } }
I20260929 01:12:13.058136 24747 sys_catalog.cc:564] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:13.060552 24753 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cae2d6cbbc2f466ba99306c47663044e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae2d6cbbc2f466ba99306c47663044e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42869 } } }
I20260929 01:12:13.061151 24753 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:13.060650 24754 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae2d6cbbc2f466ba99306c47663044e. Latest consensus state: current_term: 1 leader_uuid: "cae2d6cbbc2f466ba99306c47663044e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae2d6cbbc2f466ba99306c47663044e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42869 } } }
I20260929 01:12:13.061661 24754 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e [sys.catalog]: This master's current role is: LEADER
W20260929 01:12:13.071703 24768 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:12:13.071864 24768 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:12:13.072041 24761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:13.073885 24761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:13.081974 24761 catalog_manager.cc:1347] Generated new cluster ID: 371e914b1c9845f992af4183a7c116a9
I20260929 01:12:13.082091 24761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:13.097463 24761 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:13.098785 24761 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:13.104666 24761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e: Generated new TSK 0
I20260929 01:12:13.105580 24761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:13.118368 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42869
--builtin_ntp_servers=127.5.198.212:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:12:13.471414 24771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:13.471676 24771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:13.471837 24771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:13.479005 24771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:13.479105 24771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:13.479279 24771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:12:13.488282 24771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:13.489626 24771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:13.491466 24771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:13.504351 24776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:13.504489 24779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:13.504443 24777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:13.505947 24771 server_base.cc:1034] running on GCE node
I20260929 01:12:13.506604 24771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:13.507522 24771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:13.508777 24771 hybrid_clock.cc:648] HybridClock initialized: now 1790644333508727 us; error 58 us; skew 500 ppm
I20260929 01:12:13.509130 24771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:12:13.511411 24771 webserver.cc:460] Webserver started at http://127.5.198.193:40463/ using document root <none> and password file <none>
I20260929 01:12:13.512100 24771 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:13.512231 24771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:13.512634 24771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:13.515506 24771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "a6a1ff532eeb432daeab96a09de76ac2"
format_stamp: "Formatted at 2026-09-29 01:12:13 on dist-test-slave-41cl"
server_key: "b2c590cffe376a85513efcfb3811d0bf"
server_key_iv: "ea473f79e70e75afd3c0d52a0910c33f"
server_key_version: "encryptionkey@0"
I20260929 01:12:13.516261 24771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "a6a1ff532eeb432daeab96a09de76ac2"
format_stamp: "Formatted at 2026-09-29 01:12:13 on dist-test-slave-41cl"
server_key: "b2c590cffe376a85513efcfb3811d0bf"
server_key_iv: "ea473f79e70e75afd3c0d52a0910c33f"
server_key_version: "encryptionkey@0"
I20260929 01:12:13.521277 24771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:12:13.524693 24785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:13.525982 24771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:12:13.526147 24771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "a6a1ff532eeb432daeab96a09de76ac2"
format_stamp: "Formatted at 2026-09-29 01:12:13 on dist-test-slave-41cl"
server_key: "b2c590cffe376a85513efcfb3811d0bf"
server_key_iv: "ea473f79e70e75afd3c0d52a0910c33f"
server_key_version: "encryptionkey@0"
I20260929 01:12:13.526360 24771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:13.548803 24771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:13.549600 24771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:13.549899 24771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:13.550881 24771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:13.552416 24771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:13.552565 24771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:13.552652 24771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:13.552708 24771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:13.587185 24771 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:37067
I20260929 01:12:13.587272 24897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:37067 every 8 connection(s)
I20260929 01:12:13.588438 24771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:12:13.596736 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 24771
I20260929 01:12:13.596897 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:12:13.597340 5915 external_mini_cluster.cc:1428] Setting key 98efbae5d41d40af7b14d6d1123bfa95
I20260929 01:12:13.600059 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42869
--builtin_ntp_servers=127.5.198.212:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:12:13.601162 24900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:13.591071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:60915 (local address 127.5.198.254:42869)
0929 01:12:13.591518 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:13.591524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:13.592771 (+ 1247us) server_negotiation.cc:407] Connection header received
0929 01:12:13.594465 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:13.594470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:13.594535 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:13.594649 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:13.595643 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:13.596414 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:13.598223 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:13.598678 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:13.598776 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:13.600335 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:13.600340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:13.600344 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:13.600360 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:13.600396 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:13.600408 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:13.600533 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:13.600739 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 01:12:13.600832 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":212,"threads_started":1}
I20260929 01:12:13.602111 24901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:13.591707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:42869 (local address 127.5.198.193:60915)
0929 01:12:13.592623 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:13.592664 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:13.594221 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:13.594857 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:13.594866 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:13.594951 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:13.595485 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:13.595503 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:13.596586 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:13.596590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:13.598100 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:13.598112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:13.598898 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:13.598906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:13.599242 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:13.599929 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:12:13.599932 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:13.599968 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:12:13.600051 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:13.600527 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:13.600539 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:13.600548 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 01:12:13.600730 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 01:12:13.601102 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":292,"threads_started":1}
I20260929 01:12:13.603834 24898 heartbeater.cc:347] Connected to a master server at 127.5.198.254:42869
I20260929 01:12:13.604161 24898 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:13.604935 24898 heartbeater.cc:511] Master 127.5.198.254:42869 requested a full tablet report, sending...
I20260929 01:12:13.607165 24712 ts_manager.cc:194] Registered new tserver with Master: a6a1ff532eeb432daeab96a09de76ac2 (127.5.198.193:37067)
I20260929 01:12:13.608775 24712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.198.193:60915
W20260929 01:12:13.932088 24902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:13.932369 24902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:13.932516 24902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:13.940011 24902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:13.940115 24902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:13.940227 24902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:12:13.949548 24902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:13.950825 24902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:13.952749 24902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:13.966943 24907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:13.966920 24908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:13.966882 24910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:13.967573 24902 server_base.cc:1034] running on GCE node
I20260929 01:12:13.968211 24902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:13.969249 24902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:13.970480 24902 hybrid_clock.cc:648] HybridClock initialized: now 1790644333970438 us; error 60 us; skew 500 ppm
I20260929 01:12:13.970805 24902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:12:13.973282 24902 webserver.cc:460] Webserver started at http://127.5.198.194:41463/ using document root <none> and password file <none>
I20260929 01:12:13.973986 24902 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:13.974118 24902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:13.974448 24902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:13.977470 24902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "10c456d1b1b9454eb8dca99cc72d0639"
format_stamp: "Formatted at 2026-09-29 01:12:13 on dist-test-slave-41cl"
server_key: "fd02623d607f6dcea187af008622ccf9"
server_key_iv: "4d451a14eb935022bf55509c0a4853a2"
server_key_version: "encryptionkey@0"
I20260929 01:12:13.978268 24902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "10c456d1b1b9454eb8dca99cc72d0639"
format_stamp: "Formatted at 2026-09-29 01:12:13 on dist-test-slave-41cl"
server_key: "fd02623d607f6dcea187af008622ccf9"
server_key_iv: "4d451a14eb935022bf55509c0a4853a2"
server_key_version: "encryptionkey@0"
I20260929 01:12:13.983462 24902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:12:13.987035 24916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:13.988330 24902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:12:13.988554 24902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "10c456d1b1b9454eb8dca99cc72d0639"
format_stamp: "Formatted at 2026-09-29 01:12:13 on dist-test-slave-41cl"
server_key: "fd02623d607f6dcea187af008622ccf9"
server_key_iv: "4d451a14eb935022bf55509c0a4853a2"
server_key_version: "encryptionkey@0"
I20260929 01:12:13.988775 24902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:14.006770 24902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:14.007570 24902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:14.007872 24902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:14.008852 24902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:14.010310 24902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:14.010433 24902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:14.010572 24902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:14.010661 24902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:14.044441 24902 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:46477
I20260929 01:12:14.044567 25028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:46477 every 8 connection(s)
I20260929 01:12:14.045723 24902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:12:14.047294 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 24902
I20260929 01:12:14.047467 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:12:14.047988 5915 external_mini_cluster.cc:1428] Setting key d72848174a5547e48bad852aac08e6d3
I20260929 01:12:14.050807 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42869
--builtin_ntp_servers=127.5.198.212:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:12:14.059335 24900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:14.048513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:54021 (local address 127.5.198.254:42869)
0929 01:12:14.048707 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:14.048711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:14.050336 (+ 1625us) server_negotiation.cc:407] Connection header received
0929 01:12:14.052217 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:14.052226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:14.052314 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:14.052424 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:14.053458 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.054548 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.056159 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.056907 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.057056 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:14.058462 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:14.058466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:14.058470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:14.058488 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:14.058518 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:14.058529 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:14.058599 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:14.058929 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 01:12:14.059048 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 01:12:14.060351 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:14.049132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:42869 (local address 127.5.198.194:54021)
0929 01:12:14.050162 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:14.050213 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:14.051878 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:14.052667 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:14.052675 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:14.052760 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:14.053291 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:14.053307 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.054712 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.054717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:14.056010 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:14.056022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.057132 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.057138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:14.057331 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:14.058108 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:12:14.058114 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:14.058170 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:12:14.058279 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:14.058627 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:14.058638 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:14.058647 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 01:12:14.058870 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 01:12:14.059330 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":355,"threads_started":1}
I20260929 01:12:14.061707 25029 heartbeater.cc:347] Connected to a master server at 127.5.198.254:42869
I20260929 01:12:14.062019 25029 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:14.062721 25029 heartbeater.cc:511] Master 127.5.198.254:42869 requested a full tablet report, sending...
I20260929 01:12:14.064131 24712 ts_manager.cc:194] Registered new tserver with Master: 10c456d1b1b9454eb8dca99cc72d0639 (127.5.198.194:46477)
I20260929 01:12:14.065404 24712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.198.194:54021
W20260929 01:12:14.372047 25032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:14.372316 25032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:14.372401 25032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:14.379846 25032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:14.379953 25032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:14.380062 25032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:12:14.389281 25032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:42869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:14.390659 25032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:14.392343 25032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:14.405184 25037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:14.405249 25038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:14.405112 25040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:14.406699 25032 server_base.cc:1034] running on GCE node
I20260929 01:12:14.407228 25032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:14.408085 25032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:14.409284 25032 hybrid_clock.cc:648] HybridClock initialized: now 1790644334409250 us; error 31 us; skew 500 ppm
I20260929 01:12:14.409607 25032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:12:14.411842 25032 webserver.cc:460] Webserver started at http://127.5.198.195:41709/ using document root <none> and password file <none>
I20260929 01:12:14.412549 25032 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:14.412657 25032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:14.412985 25032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:14.415668 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "5f1d967dbf424bd78eb10ce63755cdc3"
format_stamp: "Formatted at 2026-09-29 01:12:14 on dist-test-slave-41cl"
server_key: "5dfffe00dd4b083875953ef26786047f"
server_key_iv: "bb481ac18a888b3d14e875d5bc073b76"
server_key_version: "encryptionkey@0"
I20260929 01:12:14.416427 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "5f1d967dbf424bd78eb10ce63755cdc3"
format_stamp: "Formatted at 2026-09-29 01:12:14 on dist-test-slave-41cl"
server_key: "5dfffe00dd4b083875953ef26786047f"
server_key_iv: "bb481ac18a888b3d14e875d5bc073b76"
server_key_version: "encryptionkey@0"
I20260929 01:12:14.421341 25032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:12:14.424881 25046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:14.426198 25032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:12:14.426378 25032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "5f1d967dbf424bd78eb10ce63755cdc3"
format_stamp: "Formatted at 2026-09-29 01:12:14 on dist-test-slave-41cl"
server_key: "5dfffe00dd4b083875953ef26786047f"
server_key_iv: "bb481ac18a888b3d14e875d5bc073b76"
server_key_version: "encryptionkey@0"
I20260929 01:12:14.426625 25032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:14.442129 25032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:14.442966 25032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:14.443275 25032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:14.444244 25032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:14.445770 25032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:14.445894 25032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:14.446045 25032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:14.446134 25032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:14.480160 25032 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:38821
I20260929 01:12:14.480248 25158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:38821 every 8 connection(s)
I20260929 01:12:14.481465 25032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:12:14.488226 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 25032
I20260929 01:12:14.488406 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:12:14.488883 5915 external_mini_cluster.cc:1428] Setting key 77d5d42af76122125fbf14d84dac2e55
I20260929 01:12:14.494488 24900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:14.484019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:55609 (local address 127.5.198.254:42869)
0929 01:12:14.484265 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:14.484271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:14.485694 (+ 1423us) server_negotiation.cc:407] Connection header received
0929 01:12:14.487385 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:14.487390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:14.487488 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:14.487582 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:14.488730 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.489575 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.491786 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.492208 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.492342 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:14.493698 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:14.493703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:14.493706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:14.493723 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:14.493754 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:14.493766 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:14.493860 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:14.494083 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 01:12:14.494182 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 01:12:14.495402 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:14.484659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:42869 (local address 127.5.198.195:55609)
0929 01:12:14.485520 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:14.485561 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:14.487103 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:14.487775 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:14.487784 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:14.487887 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:14.488547 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:14.488572 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.489727 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.489732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:14.491640 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:14.491652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.492423 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.492430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:14.492687 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:14.493383 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:12:14.493385 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:14.493423 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:12:14.493496 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:14.493860 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:14.493873 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:14.493883 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 01:12:14.494057 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 01:12:14.494432 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":282,"threads_started":1}
I20260929 01:12:14.496727 25159 heartbeater.cc:347] Connected to a master server at 127.5.198.254:42869
I20260929 01:12:14.497056 25159 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:14.497799 25159 heartbeater.cc:511] Master 127.5.198.254:42869 requested a full tablet report, sending...
I20260929 01:12:14.499078 24712 ts_manager.cc:194] Registered new tserver with Master: 5f1d967dbf424bd78eb10ce63755cdc3 (127.5.198.195:38821)
I20260929 01:12:14.499862 24712 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.198.195:55609
I20260929 01:12:14.503813 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:12:14.513974 24900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:14.507903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.5.198.254:42869)
0929 01:12:14.508146 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:14.508154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:14.508455 (+ 301us) server_negotiation.cc:407] Connection header received
0929 01:12:14.508699 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:14.508703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:14.508776 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:14.508880 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:14.509379 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.510069 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.511485 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.511904 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.512041 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:14.513223 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:14.513228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:14.513231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:14.513248 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:14.513277 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:14.513288 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:14.513442 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:14.513651 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 01:12:14.513707 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:12:14.517791 24712 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:12:14.524537 24712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:12:14.548766 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:14.536745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:46477 (local address 127.0.0.1:47776)
0929 01:12:14.537795 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:14.537840 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:14.538001 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:14.539236 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:14.539243 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:14.539270 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:14.541018 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:14.541038 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.543321 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.543331 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:14.545545 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:14.545560 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.547747 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.547754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:14.547983 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:14.548033 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 01:12:14.548161 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 01:12:14.548365 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":1994,"threads_started":1}
I20260929 01:12:14.549196 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:14.536901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47776 (local address 127.5.198.194:46477)
0929 01:12:14.538590 (+ 1689us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:14.538603 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:14.538635 (+ 32us) server_negotiation.cc:407] Connection header received
0929 01:12:14.538737 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:14.538746 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:14.539004 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:14.541298 (+ 2294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:14.541582 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.543185 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.545792 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.547195 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.547330 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:14.547623 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:14.548259 (+ 636us) server_negotiation.cc:299] Negotiation successful
0929 01:12:14.548445 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1503,"thread_start_us":1390,"threads_started":1}
I20260929 01:12:14.549801 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:14.538593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.5.198.193:37067)
0929 01:12:14.540910 (+ 2317us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:14.540918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:14.541641 (+ 723us) server_negotiation.cc:407] Connection header received
0929 01:12:14.541775 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:14.541784 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:14.542047 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:14.542216 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:14.543168 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.544228 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.546971 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.548432 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.548598 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:14.548836 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:14.549306 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 01:12:14.549469 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2140,"thread_start_us":2024,"threads_started":1}
I20260929 01:12:14.550303 25175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:14.538055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37067 (local address 127.0.0.1:43236)
0929 01:12:14.541368 (+ 3313us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:14.541386 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:14.541504 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:14.542447 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:14.542452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:14.542476 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:14.542984 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:14.542995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.544369 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.544375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:14.546739 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:14.546756 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.548971 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.548977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:14.549125 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:14.549148 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 01:12:14.549264 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 01:12:14.549389 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3193,"mutex_wait_us":1,"thread_start_us":3051,"threads_started":1}
I20260929 01:12:14.550876 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:14.536745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:38821 (local address 127.0.0.1:42952)
0929 01:12:14.539012 (+ 2267us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:14.539035 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:14.539186 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:14.542434 (+ 3248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:14.542439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:14.542461 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:14.542996 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:14.543010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.544998 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.545003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:14.547134 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:14.547143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.549249 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.549424 (+ 175us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:14.549622 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:14.549654 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 01:12:14.549730 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 01:12:14.549853 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2059,"mutex_wait_us":216,"thread_start_us":302,"threads_started":1}
I20260929 01:12:14.551746 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:14.537286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.5.198.195:38821)
0929 01:12:14.541833 (+ 4547us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:14.541842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:14.541879 (+ 37us) server_negotiation.cc:407] Connection header received
0929 01:12:14.541986 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:14.541997 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:14.542230 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:14.542925 (+ 695us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:14.543239 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.544847 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.547424 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.548526 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.548662 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:14.551263 (+ 2601us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:14.551334 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 01:12:14.551468 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4366,"thread_start_us":4187,"threads_started":1}
I20260929 01:12:14.553411 24964 tablet_service.cc:1467] Processing CreateTablet for tablet fb73efe00a68445f834819cc07c6abcc (DEFAULT_TABLE table=test-table [id=752000ca8dfb4e198ca395e3bf6d4ce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:14.553937 24833 tablet_service.cc:1467] Processing CreateTablet for tablet fb73efe00a68445f834819cc07c6abcc (DEFAULT_TABLE table=test-table [id=752000ca8dfb4e198ca395e3bf6d4ce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:14.555030 24964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb73efe00a68445f834819cc07c6abcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:14.554757 25094 tablet_service.cc:1467] Processing CreateTablet for tablet fb73efe00a68445f834819cc07c6abcc (DEFAULT_TABLE table=test-table [id=752000ca8dfb4e198ca395e3bf6d4ce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:14.556318 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb73efe00a68445f834819cc07c6abcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:14.556620 24833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb73efe00a68445f834819cc07c6abcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:14.568819 25177 tablet_bootstrap.cc:492] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3: Bootstrap starting.
I20260929 01:12:14.573369 25178 tablet_bootstrap.cc:492] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639: Bootstrap starting.
I20260929 01:12:14.573724 25177 tablet_bootstrap.cc:654] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:14.575196 25177 log.cc:824] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:14.576826 25179 tablet_bootstrap.cc:492] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2: Bootstrap starting.
I20260929 01:12:14.577365 25178 tablet_bootstrap.cc:654] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:14.578584 25177 tablet_bootstrap.cc:492] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3: No bootstrap required, opened a new log
I20260929 01:12:14.578689 25178 log.cc:824] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:14.578864 25177 ts_tablet_manager.cc:1397] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260929 01:12:14.580807 25179 tablet_bootstrap.cc:654] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:14.582111 25179 log.cc:824] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:14.582209 25178 tablet_bootstrap.cc:492] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639: No bootstrap required, opened a new log
I20260929 01:12:14.582464 25178 ts_tablet_manager.cc:1397] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260929 01:12:14.585171 25179 tablet_bootstrap.cc:492] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2: No bootstrap required, opened a new log
I20260929 01:12:14.585450 25179 ts_tablet_manager.cc:1397] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260929 01:12:14.586993 25177 raft_consensus.cc:348] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c456d1b1b9454eb8dca99cc72d0639" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46477 } } peers { permanent_uuid: "5f1d967dbf424bd78eb10ce63755cdc3" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38821 } } peers { permanent_uuid: "a6a1ff532eeb432daeab96a09de76ac2" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37067 } }
I20260929 01:12:14.587433 25177 raft_consensus.cc:374] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:14.587528 25177 raft_consensus.cc:729] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1d967dbf424bd78eb10ce63755cdc3, State: Initialized, Role: FOLLOWER
I20260929 01:12:14.588232 25177 consensus_queue.cc:260] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c456d1b1b9454eb8dca99cc72d0639" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46477 } } peers { permanent_uuid: "5f1d967dbf424bd78eb10ce63755cdc3" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38821 } } peers { permanent_uuid: "a6a1ff532eeb432daeab96a09de76ac2" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37067 } }
I20260929 01:12:14.591292 25159 heartbeater.cc:503] Master 127.5.198.254:42869 was elected leader, sending a full tablet report...
I20260929 01:12:14.591953 25177 ts_tablet_manager.cc:1428] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260929 01:12:14.593649 25178 raft_consensus.cc:348] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c456d1b1b9454eb8dca99cc72d0639" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46477 } } peers { permanent_uuid: "5f1d967dbf424bd78eb10ce63755cdc3" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38821 } } peers { permanent_uuid: "a6a1ff532eeb432daeab96a09de76ac2" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37067 } }
I20260929 01:12:14.594066 25178 raft_consensus.cc:374] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:14.594161 25178 raft_consensus.cc:729] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c456d1b1b9454eb8dca99cc72d0639, State: Initialized, Role: FOLLOWER
I20260929 01:12:14.594868 25178 consensus_queue.cc:260] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c456d1b1b9454eb8dca99cc72d0639" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46477 } } peers { permanent_uuid: "5f1d967dbf424bd78eb10ce63755cdc3" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38821 } } peers { permanent_uuid: "a6a1ff532eeb432daeab96a09de76ac2" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37067 } }
I20260929 01:12:14.598462 25179 raft_consensus.cc:348] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c456d1b1b9454eb8dca99cc72d0639" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46477 } } peers { permanent_uuid: "5f1d967dbf424bd78eb10ce63755cdc3" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38821 } } peers { permanent_uuid: "a6a1ff532eeb432daeab96a09de76ac2" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37067 } }
I20260929 01:12:14.598841 25179 raft_consensus.cc:374] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:14.598960 25179 raft_consensus.cc:729] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a1ff532eeb432daeab96a09de76ac2, State: Initialized, Role: FOLLOWER
I20260929 01:12:14.599560 25179 consensus_queue.cc:260] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c456d1b1b9454eb8dca99cc72d0639" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46477 } } peers { permanent_uuid: "5f1d967dbf424bd78eb10ce63755cdc3" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38821 } } peers { permanent_uuid: "a6a1ff532eeb432daeab96a09de76ac2" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37067 } }
I20260929 01:12:14.600663 24898 heartbeater.cc:503] Master 127.5.198.254:42869 was elected leader, sending a full tablet report...
I20260929 01:12:14.601346 25179 ts_tablet_manager.cc:1428] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260929 01:12:14.602344 25029 heartbeater.cc:503] Master 127.5.198.254:42869 was elected leader, sending a full tablet report...
I20260929 01:12:14.602798 25178 ts_tablet_manager.cc:1428] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639: Time spent starting tablet: real 0.020s user 0.018s sys 0.002s
W20260929 01:12:14.733263 25160 tablet.cc:2369] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:12:14.797727 25030 tablet.cc:2369] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:12:14.840966 24899 tablet.cc:2369] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:12:14.889878 25184 raft_consensus.cc:482] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:12:14.890303 25184 raft_consensus.cc:504] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c456d1b1b9454eb8dca99cc72d0639" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46477 } } peers { permanent_uuid: "5f1d967dbf424bd78eb10ce63755cdc3" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38821 } } peers { permanent_uuid: "a6a1ff532eeb432daeab96a09de76ac2" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37067 } }
I20260929 01:12:14.892046 25184 leader_election.cc:288] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f1d967dbf424bd78eb10ce63755cdc3 (127.5.198.195:38821), a6a1ff532eeb432daeab96a09de76ac2 (127.5.198.193:37067)
I20260929 01:12:14.899581 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:14.892730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:35999 (local address 127.5.198.195:38821)
0929 01:12:14.892903 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:14.892908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:14.893517 (+ 609us) server_negotiation.cc:407] Connection header received
0929 01:12:14.893735 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:14.893743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:14.893838 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:14.893943 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:14.894642 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.895572 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.897798 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.898593 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.898766 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:14.898849 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:14.899194 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 01:12:14.899330 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:12:14.899602 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:14.893715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:34813 (local address 127.5.198.193:37067)
0929 01:12:14.893876 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:14.893880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:14.894195 (+ 315us) server_negotiation.cc:407] Connection header received
0929 01:12:14.894330 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:14.894335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:14.894430 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:14.894546 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:14.895175 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.896040 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.897874 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.898629 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.898810 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:14.898882 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:14.899224 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 01:12:14.899346 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 01:12:14.899835 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:14.893512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37067 (local address 127.5.198.194:34813)
0929 01:12:14.894095 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:14.894108 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:14.894219 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:14.894560 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:14.894580 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:14.894605 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:14.894995 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:14.895007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.896212 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.896216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:14.897649 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:14.897664 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.898860 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.898869 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:14.899035 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:14.899074 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 01:12:14.899185 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 01:12:14.899337 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":46,"thread_start_us":380,"threads_started":1}
I20260929 01:12:14.899930 25186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:14.892688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:38821 (local address 127.5.198.194:35999)
0929 01:12:14.893366 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:14.893382 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:14.893567 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:14.893989 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:14.893994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:14.894023 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:14.894495 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:14.894511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.895717 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.895723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:14.897612 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:14.897625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:14.898805 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:14.898811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:14.899025 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:14.899064 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 01:12:14.899137 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 01:12:14.899302 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":129,"spinlock_wait_cycles":24320,"thread_start_us":376,"threads_started":1}
I20260929 01:12:14.900951 24853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb73efe00a68445f834819cc07c6abcc" candidate_uuid: "10c456d1b1b9454eb8dca99cc72d0639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a1ff532eeb432daeab96a09de76ac2" is_pre_election: true
I20260929 01:12:14.901124 25114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb73efe00a68445f834819cc07c6abcc" candidate_uuid: "10c456d1b1b9454eb8dca99cc72d0639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d967dbf424bd78eb10ce63755cdc3" is_pre_election: true
I20260929 01:12:14.901525 24853 raft_consensus.cc:2454] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c456d1b1b9454eb8dca99cc72d0639 in term 0.
I20260929 01:12:14.901652 25114 raft_consensus.cc:2454] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10c456d1b1b9454eb8dca99cc72d0639 in term 0.
I20260929 01:12:14.902243 24918 leader_election.cc:302] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10c456d1b1b9454eb8dca99cc72d0639, a6a1ff532eeb432daeab96a09de76ac2; no voters:
I20260929 01:12:14.902619 25184 raft_consensus.cc:2781] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:12:14.902773 25184 raft_consensus.cc:482] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:12:14.902874 25184 raft_consensus.cc:3037] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:14.905081 25184 raft_consensus.cc:504] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c456d1b1b9454eb8dca99cc72d0639" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46477 } } peers { permanent_uuid: "5f1d967dbf424bd78eb10ce63755cdc3" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38821 } } peers { permanent_uuid: "a6a1ff532eeb432daeab96a09de76ac2" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37067 } }
I20260929 01:12:14.905767 25184 leader_election.cc:288] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [CANDIDATE]: Term 1 election: Requested vote from peers 5f1d967dbf424bd78eb10ce63755cdc3 (127.5.198.195:38821), a6a1ff532eeb432daeab96a09de76ac2 (127.5.198.193:37067)
I20260929 01:12:14.906230 25114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb73efe00a68445f834819cc07c6abcc" candidate_uuid: "10c456d1b1b9454eb8dca99cc72d0639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1d967dbf424bd78eb10ce63755cdc3"
I20260929 01:12:14.906276 24853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb73efe00a68445f834819cc07c6abcc" candidate_uuid: "10c456d1b1b9454eb8dca99cc72d0639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a1ff532eeb432daeab96a09de76ac2"
I20260929 01:12:14.906415 25114 raft_consensus.cc:3037] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:14.906523 24853 raft_consensus.cc:3037] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:14.909066 25114 raft_consensus.cc:2454] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c456d1b1b9454eb8dca99cc72d0639 in term 1.
I20260929 01:12:14.909313 24853 raft_consensus.cc:2454] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10c456d1b1b9454eb8dca99cc72d0639 in term 1.
I20260929 01:12:14.909478 24918 leader_election.cc:302] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10c456d1b1b9454eb8dca99cc72d0639, 5f1d967dbf424bd78eb10ce63755cdc3; no voters:
I20260929 01:12:14.909757 25184 raft_consensus.cc:2781] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:14.910182 25184 raft_consensus.cc:686] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [term 1 LEADER]: Becoming Leader. State: Replica: 10c456d1b1b9454eb8dca99cc72d0639, State: Running, Role: LEADER
I20260929 01:12:14.910661 25184 consensus_queue.cc:237] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c456d1b1b9454eb8dca99cc72d0639" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46477 } } peers { permanent_uuid: "5f1d967dbf424bd78eb10ce63755cdc3" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38821 } } peers { permanent_uuid: "a6a1ff532eeb432daeab96a09de76ac2" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37067 } }
I20260929 01:12:14.915035 24712 catalog_manager.cc:5495] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10c456d1b1b9454eb8dca99cc72d0639 (127.5.198.194). New cstate: current_term: 1 leader_uuid: "10c456d1b1b9454eb8dca99cc72d0639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c456d1b1b9454eb8dca99cc72d0639" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 46477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1d967dbf424bd78eb10ce63755cdc3" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a1ff532eeb432daeab96a09de76ac2" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37067 } health_report { overall_health: UNKNOWN } } }
I20260929 01:12:15.010054 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.004283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47782 (local address 127.5.198.194:46477)
0929 01:12:15.004665 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:15.004670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:15.004688 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:12:15.004776 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:15.004780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:15.004853 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:15.004975 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:12:15.005437 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:15.006323 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:15.007702 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:15.008102 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:15.008210 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:15.008876 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:12:15.009410 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:12:15.009555 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:15.009715 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 01:12:15.009776 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266}
I20260929 01:12:15.018977 24853 raft_consensus.cc:1261] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2 [term 1 FOLLOWER]: Refusing update from remote peer 10c456d1b1b9454eb8dca99cc72d0639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:12:15.018977 25114 raft_consensus.cc:1261] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3 [term 1 FOLLOWER]: Refusing update from remote peer 10c456d1b1b9454eb8dca99cc72d0639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:12:15.020105 25188 consensus_queue.cc:1035] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a1ff532eeb432daeab96a09de76ac2" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:12:15.020545 25184 consensus_queue.cc:1035] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1d967dbf424bd78eb10ce63755cdc3" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 38821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:12:15.041482 25192 mvcc.cc:205] Tried to move back new op lower bound from 7334479196225081344 to 7334479195797741568. Current Snapshot: MvccSnapshot[applied={T|T < 7334479196225081344}]
I20260929 01:12:15.055208 24712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:12:15.055610 24712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:12:15.060707 24712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cae2d6cbbc2f466ba99306c47663044e: Sending DeleteTablet for 3 replicas of tablet fb73efe00a68445f834819cc07c6abcc
I20260929 01:12:15.062158 24964 tablet_service.cc:1514] Processing DeleteTablet for tablet fb73efe00a68445f834819cc07c6abcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:15 UTC) from {username='slave'} at 127.0.0.1:47776
I20260929 01:12:15.063246 25094 tablet_service.cc:1514] Processing DeleteTablet for tablet fb73efe00a68445f834819cc07c6abcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:15 UTC) from {username='slave'} at 127.0.0.1:42952
I20260929 01:12:15.063651 25200 tablet_replica.cc:330] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639: stopping tablet replica
I20260929 01:12:15.064100 25200 raft_consensus.cc:2229] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:12:15.064728 24832 tablet_service.cc:1514] Processing DeleteTablet for tablet fb73efe00a68445f834819cc07c6abcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:15 UTC) from {username='slave'} at 127.0.0.1:43236
I20260929 01:12:15.064728 24833 tablet_service.cc:1514] Processing DeleteTablet for tablet fb73efe00a68445f834819cc07c6abcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:15 UTC) from {username='slave'} at 127.0.0.1:43236
I20260929 01:12:15.065299 25200 raft_consensus.cc:2258] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:15.066039 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 24771
I20260929 01:12:15.066088 25201 tablet_replica.cc:330] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3: stopping tablet replica
I20260929 01:12:15.066494 25201 raft_consensus.cc:2229] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:12:15.067001 25201 raft_consensus.cc:2258] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:15.069347 25201 ts_tablet_manager.cc:1905] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:15.077018 25202 tablet_replica.cc:330] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2: stopping tablet replica
I20260929 01:12:15.077473 25202 raft_consensus.cc:2229] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:12:15.078027 25202 raft_consensus.cc:2258] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:15.079077 25200 ts_tablet_manager.cc:1905] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:15.081959 25202 ts_tablet_manager.cc:1905] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:15.084239 25201 ts_tablet_manager.cc:1918] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:12:15.084507 25201 log.cc:1196] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/fb73efe00a68445f834819cc07c6abcc
I20260929 01:12:15.085101 25201 ts_tablet_manager.cc:1939] T fb73efe00a68445f834819cc07c6abcc P 5f1d967dbf424bd78eb10ce63755cdc3: Deleting consensus metadata
I20260929 01:12:15.086971 24697 catalog_manager.cc:4841] TS 5f1d967dbf424bd78eb10ce63755cdc3 (127.5.198.195:38821): tablet fb73efe00a68445f834819cc07c6abcc (table test-table [id=752000ca8dfb4e198ca395e3bf6d4ce9]) successfully deleted
I20260929 01:12:15.087620 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.069209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.5.198.193:37067)
0929 01:12:15.069371 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:15.069377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:15.078880 (+ 9503us) server_negotiation.cc:407] Connection header received
0929 01:12:15.079210 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:15.079217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:15.079323 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:15.079409 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:15.080147 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:15.081300 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:15.083293 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:15.084063 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:15.084197 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:15.085678 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:15.085689 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:15.085706 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:15.086517 (+ 811us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:15.086588 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:15.086616 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:15.086716 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:15.087023 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 01:12:15.087157 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 01:12:15.089075 24893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:15.090167 25200 ts_tablet_manager.cc:1918] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:12:15.090380 25200 log.cc:1196] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/fb73efe00a68445f834819cc07c6abcc
I20260929 01:12:15.090900 25200 ts_tablet_manager.cc:1939] T fb73efe00a68445f834819cc07c6abcc P 10c456d1b1b9454eb8dca99cc72d0639: Deleting consensus metadata
I20260929 01:12:15.092813 24696 catalog_manager.cc:4841] TS 10c456d1b1b9454eb8dca99cc72d0639 (127.5.198.194:46477): tablet fb73efe00a68445f834819cc07c6abcc (table test-table [id=752000ca8dfb4e198ca395e3bf6d4ce9]) successfully deleted
I20260929 01:12:15.180042 25202 ts_tablet_manager.cc:1918] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:12:15.180312 25202 log.cc:1196] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/fb73efe00a68445f834819cc07c6abcc
I20260929 01:12:15.180895 25202 ts_tablet_manager.cc:1939] T fb73efe00a68445f834819cc07c6abcc P a6a1ff532eeb432daeab96a09de76ac2: Deleting consensus metadata
I20260929 01:12:15.181300 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 24771
I20260929 01:12:15.199884 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 24902
W20260929 01:12:15.200001 24697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.198.193:37067 (error 108)
I20260929 01:12:15.203344 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.202494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37067 (local address 127.0.0.1:43268)
0929 01:12:15.202994 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:15.203125 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":226,"threads_started":1}
W20260929 01:12:15.204795 24697 catalog_manager.cc:4584] TS a6a1ff532eeb432daeab96a09de76ac2 (127.5.198.193:37067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb73efe00a68445f834819cc07c6abcc: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37067: connect: Connection refused (error 111)
I20260929 01:12:15.208756 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.200697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.5.198.194:46477)
0929 01:12:15.200870 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:15.200875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:15.201452 (+ 577us) server_negotiation.cc:407] Connection header received
0929 01:12:15.201538 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:15.201547 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:15.201668 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:15.201790 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:15.202237 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:15.203306 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:15.205437 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:15.205841 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:15.205934 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:15.207218 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:15.207227 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:15.207240 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:15.207927 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:15.207973 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:15.207993 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:15.208087 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:15.208263 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 01:12:15.208340 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:12:15.209532 25024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:15.249459 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.248924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37067 (local address 127.0.0.1:43284)
0929 01:12:15.249178 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:15.249287 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 01:12:15.250643 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.250220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37067 (local address 127.0.0.1:43290)
0929 01:12:15.250375 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:15.250474 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 01:12:15.258874 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.258466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37067 (local address 127.0.0.1:43292)
0929 01:12:15.258641 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:15.258730 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 01:12:15.259864 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.259477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37067 (local address 127.0.0.1:43306)
0929 01:12:15.259625 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:15.259703 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 01:12:15.299542 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.298976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37067 (local address 127.0.0.1:43320)
0929 01:12:15.299224 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:15.299339 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 01:12:15.300776 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.300262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37067 (local address 127.0.0.1:43336)
0929 01:12:15.300518 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:15.300602 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 01:12:15.324353 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 24902
I20260929 01:12:15.336333 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.335820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37067 (local address 127.0.0.1:43342)
0929 01:12:15.336086 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:15.336186 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 01:12:15.337404 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.337019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37067 (local address 127.0.0.1:43350)
0929 01:12:15.337195 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:15.337276 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 01:12:15.344810 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 25032
I20260929 01:12:15.351652 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.345613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.5.198.195:38821)
0929 01:12:15.345814 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:15.345820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:15.345840 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:12:15.345986 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:15.345991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:15.346094 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:15.346240 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:15.346612 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:15.347338 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:15.348651 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:15.349074 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:15.349206 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:15.350273 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:15.350282 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:15.350297 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:15.350862 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:15.350907 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:15.350926 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:15.351070 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:15.351210 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 01:12:15.351322 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 01:12:15.352435 25154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:15.402020 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.401534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37067 (local address 127.0.0.1:43358)
0929 01:12:15.401777 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:15.401868 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 01:12:15.403179 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.402788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37067 (local address 127.0.0.1:43372)
0929 01:12:15.402945 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:15.403018 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 01:12:15.440418 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.439311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37067 (local address 127.0.0.1:43388)
0929 01:12:15.440133 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:15.440242 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593}
I20260929 01:12:15.441001 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 25032
I20260929 01:12:15.442956 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:15.442574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:37067 (local address 127.0.0.1:43390)
0929 01:12:15.442751 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:15.442822 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:37067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 01:12:15.454741 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 24681
I20260929 01:12:15.455899 24742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:15.567119 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 24681
2026-09-29T01:12:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3042 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:15 dist-test-slave-41cl krb5kdc[25212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:15 dist-test-slave-41cl krb5kdc[25212](info): set up 2 sockets
Sep 29 01:12:15 dist-test-slave-41cl krb5kdc[25212](info): commencing operation
krb5kdc: starting...
W20260929 01:12:16.684773 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:16 dist-test-slave-41cl krb5kdc[25212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:12:16.936909 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:37215
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:37215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:17.260268 25225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:17.260646 25225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:17.260823 25225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:17.267876 25225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:17.268023 25225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:17.268118 25225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:17.268203 25225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:17.268270 25225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:17.278019 25225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:37215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:37215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:17.279289 25225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:17.280658 25225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:17.292698 25233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:17.292620 25231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:17.292636 25230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:17.294235 25225 server_base.cc:1034] running on GCE node
I20260929 01:12:17.295193 25225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:17.296841 25225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:17.298090 25225 hybrid_clock.cc:648] HybridClock initialized: now 1790644337298049 us; error 59 us; skew 500 ppm
I20260929 01:12:17.298456 25225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:17 dist-test-slave-41cl krb5kdc[25212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644337, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:17.305943 25225 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:12:17.307420 25225 webserver.cc:460] Webserver started at http://127.5.198.254:39605/ using document root <none> and password file <none>
I20260929 01:12:17.308125 25225 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:17.308274 25225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:17.308661 25225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:17.311650 25225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "30f9082108174ac480495aecb0b37518"
format_stamp: "Formatted at 2026-09-29 01:12:17 on dist-test-slave-41cl"
server_key: "f5d1391b72c41f8eab0972130bf66577"
server_key_iv: "cd7476b626273e4810b335b3d83abb75"
server_key_version: "encryptionkey@0"
I20260929 01:12:17.312392 25225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "30f9082108174ac480495aecb0b37518"
format_stamp: "Formatted at 2026-09-29 01:12:17 on dist-test-slave-41cl"
server_key: "f5d1391b72c41f8eab0972130bf66577"
server_key_iv: "cd7476b626273e4810b335b3d83abb75"
server_key_version: "encryptionkey@0"
I20260929 01:12:17.317458 25225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 01:12:17.320948 25240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:17.322357 25225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:12:17.322608 25225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "30f9082108174ac480495aecb0b37518"
format_stamp: "Formatted at 2026-09-29 01:12:17 on dist-test-slave-41cl"
server_key: "f5d1391b72c41f8eab0972130bf66577"
server_key_iv: "cd7476b626273e4810b335b3d83abb75"
server_key_version: "encryptionkey@0"
I20260929 01:12:17.322809 25225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:17.335637 25225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:17.339659 25225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:17.339936 25225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:17.358556 25225 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:37215
I20260929 01:12:17.358584 25291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:37215 every 8 connection(s)
I20260929 01:12:17.359936 25225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:17.363617 25292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:17.369504 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 25225
I20260929 01:12:17.369701 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:17.370242 5915 external_mini_cluster.cc:1428] Setting key dffb133158ee35a48123583921dc4f5d
I20260929 01:12:17.374748 25292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518: Bootstrap starting.
I20260929 01:12:17.378400 25292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:17.379645 25292 log.cc:824] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:17.382830 25292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518: No bootstrap required, opened a new log
Sep 29 01:12:17 dist-test-slave-41cl krb5kdc[25212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:12:17.390829 25292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f9082108174ac480495aecb0b37518" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37215 } }
I20260929 01:12:17.391172 25292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:17.391295 25292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f9082108174ac480495aecb0b37518, State: Initialized, Role: FOLLOWER
I20260929 01:12:17.391904 25292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f9082108174ac480495aecb0b37518" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37215 } }
I20260929 01:12:17.392156 25292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:17.392294 25292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:17.392539 25292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:17.393821 25295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:17.372125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53930 (local address 127.5.198.254:37215)
0929 01:12:17.372977 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:17.372989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:17.373086 (+ 97us) server_negotiation.cc:407] Connection header received
0929 01:12:17.374837 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:17.374868 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:17.375262 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:17.375574 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:17.377413 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:17.378799 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:17.380638 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:17.381435 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:17.381566 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:17.386297 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:17.386304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:17.386318 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:17.386356 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:17.389517 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:17.390271 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:17.390278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:17.390283 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:17.390435 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:17.390844 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:17.390848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:17.390850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:17.391499 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:17.391806 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:17.392230 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 01:12:17.392818 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":433,"threads_started":1}
I20260929 01:12:17.394898 25292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f9082108174ac480495aecb0b37518" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37215 } }
I20260929 01:12:17.395386 25292 leader_election.cc:302] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30f9082108174ac480495aecb0b37518; no voters:
I20260929 01:12:17.395985 25292 leader_election.cc:288] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:17.396136 25297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:17.398371 25292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:17.398584 25297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [term 1 LEADER]: Becoming Leader. State: Replica: 30f9082108174ac480495aecb0b37518, State: Running, Role: LEADER
I20260929 01:12:17.399133 25297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f9082108174ac480495aecb0b37518" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37215 } }
I20260929 01:12:17.402014 25299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30f9082108174ac480495aecb0b37518. Latest consensus state: current_term: 1 leader_uuid: "30f9082108174ac480495aecb0b37518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f9082108174ac480495aecb0b37518" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37215 } } }
I20260929 01:12:17.402020 25298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30f9082108174ac480495aecb0b37518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f9082108174ac480495aecb0b37518" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 37215 } } }
I20260929 01:12:17.402472 25298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:17.402474 25299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:17.408764 25306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:17.412896 25306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:17.421082 25306 catalog_manager.cc:1347] Generated new cluster ID: 43e2653561ea4a469c6fb79a0eada465
I20260929 01:12:17.421195 25306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:17.451020 25306 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:17.452925 25306 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:17.467198 25306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518: Generated new TSK 0
I20260929 01:12:17.468091 25306 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
Entry for principal kudu/127.5.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab.
I20260929 01:12:17.529539 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37215
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:17.859623 25318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:17.859953 25318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:17.860200 25318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:17.869344 25318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:17.869455 25318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:17.869669 25318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:12:17.880609 25318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.193.keytab
--principal=kudu/127.5.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:17.882014 25318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:17.883494 25318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:17.897632 25323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:17.897643 25324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:17.898005 25326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:17.898381 25318 server_base.cc:1034] running on GCE node
I20260929 01:12:17.899083 25318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:17.900063 25318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:17.901316 25318 hybrid_clock.cc:648] HybridClock initialized: now 1790644337901258 us; error 51 us; skew 500 ppm
I20260929 01:12:17.901679 25318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:17 dist-test-slave-41cl krb5kdc[25212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644337, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:17.905340 25318 init.cc:377] Logged in from keytab as kudu/127.5.198.193@KRBTEST.COM (short username kudu)
I20260929 01:12:17.906911 25318 webserver.cc:460] Webserver started at http://127.5.198.193:34515/ using document root <none> and password file <none>
I20260929 01:12:17.907630 25318 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:17.907790 25318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:17.908151 25318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:17.911199 25318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "0e0503c99acd4c8089ddd3a141a4648d"
format_stamp: "Formatted at 2026-09-29 01:12:17 on dist-test-slave-41cl"
server_key: "e02eddc17e0795b1fe9b240054ce676b"
server_key_iv: "248cabc8bb0e920c6d927d9e2d62eb0f"
server_key_version: "encryptionkey@0"
I20260929 01:12:17.911978 25318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "0e0503c99acd4c8089ddd3a141a4648d"
format_stamp: "Formatted at 2026-09-29 01:12:17 on dist-test-slave-41cl"
server_key: "e02eddc17e0795b1fe9b240054ce676b"
server_key_iv: "248cabc8bb0e920c6d927d9e2d62eb0f"
server_key_version: "encryptionkey@0"
I20260929 01:12:17.917194 25318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 01:12:17.920806 25333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:17.922223 25318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:12:17.922410 25318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "0e0503c99acd4c8089ddd3a141a4648d"
format_stamp: "Formatted at 2026-09-29 01:12:17 on dist-test-slave-41cl"
server_key: "e02eddc17e0795b1fe9b240054ce676b"
server_key_iv: "248cabc8bb0e920c6d927d9e2d62eb0f"
server_key_version: "encryptionkey@0"
I20260929 01:12:17.922657 25318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:17.940944 25318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:17.945384 25318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:17.945678 25318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:17.946626 25318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:17.947978 25318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:17.948092 25318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:17.948237 25318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:17.948310 25318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:17.982494 25318 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:35945
I20260929 01:12:17.982592 25445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:35945 every 8 connection(s)
I20260929 01:12:17.983906 25318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:12:17.993454 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 25318
I20260929 01:12:17.993631 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:12:17.994074 5915 external_mini_cluster.cc:1428] Setting key ca04f7eb542dbf9bd4b10e2a7ee44d41
Sep 29 01:12:17 dist-test-slave-41cl krb5kdc[25212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644337, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.193@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:12:18.006050 25448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:17.986858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:56281 (local address 127.5.198.254:37215)
0929 01:12:17.987366 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:17.987371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:17.988258 (+ 887us) server_negotiation.cc:407] Connection header received
0929 01:12:17.990342 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:17.990349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:17.990456 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:17.990590 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:17.992232 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:17.993079 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:17.995033 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:17.995492 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:17.995641 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:18.000165 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:18.000170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:18.000173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:18.000206 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:18.002914 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:18.003710 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:18.003716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:18.003719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:18.003855 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:18.004282 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:18.004289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:18.004292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:18.004619 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:18.004788 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:18.005449 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 01:12:18.005690 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":270,"threads_started":1}
I20260929 01:12:18.006970 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:17.987307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37215 (local address 127.5.198.193:56281)
0929 01:12:17.988073 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:17.988123 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:17.990044 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:17.990900 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:17.990909 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:17.991457 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:17.992034 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:17.992053 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:17.993246 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:17.993253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:17.994866 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:17.994879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:17.995751 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:17.995761 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:17.996100 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:17.997030 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:12:17.997075 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:17.999937 (+ 2862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:18.003056 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:18.003065 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:18.003090 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:18.003583 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:18.003967 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:18.003973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:18.003976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:18.004169 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:18.004740 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:18.004749 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:18.005090 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 01:12:18.005359 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 01:12:18.005857 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":308,"threads_started":1}
I20260929 01:12:18.008736 25446 heartbeater.cc:347] Connected to a master server at 127.5.198.254:37215
I20260929 01:12:18.009033 25446 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:18.009773 25446 heartbeater.cc:511] Master 127.5.198.254:37215 requested a full tablet report, sending...
I20260929 01:12:18.012406 25257 ts_manager.cc:194] Registered new tserver with Master: 0e0503c99acd4c8089ddd3a141a4648d (127.5.198.193:35945)
I20260929 01:12:18.014755 25257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.193@KRBTEST.COM'} at 127.5.198.193:56281
WARNING: no policy specified for kudu/127.5.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
Entry for principal kudu/127.5.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab.
I20260929 01:12:18.052012 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37215
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:18.383958 25452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:18.384231 25452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:18.384325 25452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:18.391604 25452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:18.391712 25452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:18.391820 25452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:12:18.401674 25452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.194.keytab
--principal=kudu/127.5.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:18.402951 25452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:18.404215 25452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:18.417054 25457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:18.417014 25458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:18.417107 25460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:18.418143 25452 server_base.cc:1034] running on GCE node
I20260929 01:12:18.418782 25452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:18.419667 25452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:18.420864 25452 hybrid_clock.cc:648] HybridClock initialized: now 1790644338420802 us; error 53 us; skew 500 ppm
I20260929 01:12:18.421214 25452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:18 dist-test-slave-41cl krb5kdc[25212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644338, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:18.424315 25452 init.cc:377] Logged in from keytab as kudu/127.5.198.194@KRBTEST.COM (short username kudu)
I20260929 01:12:18.425762 25452 webserver.cc:460] Webserver started at http://127.5.198.194:41889/ using document root <none> and password file <none>
I20260929 01:12:18.426539 25452 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:18.426676 25452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:18.426950 25452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:18.429903 25452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "99ef551b67d543e8b26a7d04ab8d49ef"
format_stamp: "Formatted at 2026-09-29 01:12:18 on dist-test-slave-41cl"
server_key: "2fde29fd7edc890cc968301f2e5d915e"
server_key_iv: "f8d7a875188e9b28bb31d645bdfc9aa4"
server_key_version: "encryptionkey@0"
I20260929 01:12:18.430624 25452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "99ef551b67d543e8b26a7d04ab8d49ef"
format_stamp: "Formatted at 2026-09-29 01:12:18 on dist-test-slave-41cl"
server_key: "2fde29fd7edc890cc968301f2e5d915e"
server_key_iv: "f8d7a875188e9b28bb31d645bdfc9aa4"
server_key_version: "encryptionkey@0"
I20260929 01:12:18.435648 25452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 01:12:18.439054 25467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:18.440644 25452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:12:18.440810 25452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "99ef551b67d543e8b26a7d04ab8d49ef"
format_stamp: "Formatted at 2026-09-29 01:12:18 on dist-test-slave-41cl"
server_key: "2fde29fd7edc890cc968301f2e5d915e"
server_key_iv: "f8d7a875188e9b28bb31d645bdfc9aa4"
server_key_version: "encryptionkey@0"
I20260929 01:12:18.441013 25452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:18.453347 25452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:18.457758 25452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:18.458073 25452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:18.459077 25452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:18.460523 25452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:18.460609 25452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:18.460736 25452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:18.460788 25452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:18.495517 25452 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:39837
I20260929 01:12:18.495603 25579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:39837 every 8 connection(s)
I20260929 01:12:18.496831 25452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:12:18.505051 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 25452
I20260929 01:12:18.505231 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:12:18.505791 5915 external_mini_cluster.cc:1428] Setting key 05f403d754f6a326e3421a350477bb74
Sep 29 01:12:18 dist-test-slave-41cl krb5kdc[25212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644338, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.194@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:12:18.519099 25448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:18.499449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:48443 (local address 127.5.198.254:37215)
0929 01:12:18.499647 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:18.499652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:18.500772 (+ 1120us) server_negotiation.cc:407] Connection header received
0929 01:12:18.502793 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:18.502799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:18.502894 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:18.503004 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:18.504557 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:18.505640 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:18.507752 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:18.508339 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:18.508495 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:18.511919 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:18.511928 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:18.511936 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:18.511982 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:18.515612 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:18.516416 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:18.516425 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:18.516430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:18.516625 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:18.517116 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:18.517122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:18.517127 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:18.517515 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:18.517682 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:18.518385 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 01:12:18.518652 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 01:12:18.519842 25582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:18.499866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37215 (local address 127.5.198.194:48443)
0929 01:12:18.500613 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:18.500650 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:18.502488 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:18.503247 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:18.503255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:18.503825 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:18.504367 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:18.504385 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:18.505824 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:18.505829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:18.507588 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:18.507603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:18.508619 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:18.508626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:18.508827 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:18.509531 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:12:18.509554 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:18.511686 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:18.515756 (+ 4070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:18.515763 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:18.515776 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:18.516264 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:18.516784 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:18.516788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:18.516790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:18.516990 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:18.517681 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:18.517687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:18.517994 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 01:12:18.518238 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 01:12:18.518725 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":275,"threads_started":1}
I20260929 01:12:18.521481 25580 heartbeater.cc:347] Connected to a master server at 127.5.198.254:37215
I20260929 01:12:18.521853 25580 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:18.522588 25580 heartbeater.cc:511] Master 127.5.198.254:37215 requested a full tablet report, sending...
I20260929 01:12:18.524137 25257 ts_manager.cc:194] Registered new tserver with Master: 99ef551b67d543e8b26a7d04ab8d49ef (127.5.198.194:39837)
I20260929 01:12:18.525130 25257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.194@KRBTEST.COM'} at 127.5.198.194:48443
WARNING: no policy specified for kudu/127.5.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
Entry for principal kudu/127.5.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab.
I20260929 01:12:18.564800 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37215
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:18.906064 25585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:18.906356 25585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:18.906513 25585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:18.914000 25585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:18.914110 25585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:18.914289 25585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:12:18.924036 25585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.195.keytab
--principal=kudu/127.5.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:37215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:18.925359 25585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:18.926759 25585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:18.940693 25591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:18.940730 25593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:18.941376 25590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:18.941393 25585 server_base.cc:1034] running on GCE node
I20260929 01:12:18.942049 25585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:18.943261 25585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:18.944509 25585 hybrid_clock.cc:648] HybridClock initialized: now 1790644338944439 us; error 33 us; skew 500 ppm
I20260929 01:12:18.944936 25585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:18 dist-test-slave-41cl krb5kdc[25212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644338, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:18.948042 25585 init.cc:377] Logged in from keytab as kudu/127.5.198.195@KRBTEST.COM (short username kudu)
I20260929 01:12:18.949523 25585 webserver.cc:460] Webserver started at http://127.5.198.195:39509/ using document root <none> and password file <none>
I20260929 01:12:18.950179 25585 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:18.950306 25585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:18.950619 25585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:18.953505 25585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "46ed9cdf919941b39b668ad1401a7504"
format_stamp: "Formatted at 2026-09-29 01:12:18 on dist-test-slave-41cl"
server_key: "43a4f16e214b347fd6862447bc102d55"
server_key_iv: "4e64e04364758c005f7b051f537f956c"
server_key_version: "encryptionkey@0"
I20260929 01:12:18.954250 25585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "46ed9cdf919941b39b668ad1401a7504"
format_stamp: "Formatted at 2026-09-29 01:12:18 on dist-test-slave-41cl"
server_key: "43a4f16e214b347fd6862447bc102d55"
server_key_iv: "4e64e04364758c005f7b051f537f956c"
server_key_version: "encryptionkey@0"
I20260929 01:12:18.959299 25585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 01:12:18.962715 25600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:18.964080 25585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 01:12:18.964321 25585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "46ed9cdf919941b39b668ad1401a7504"
format_stamp: "Formatted at 2026-09-29 01:12:18 on dist-test-slave-41cl"
server_key: "43a4f16e214b347fd6862447bc102d55"
server_key_iv: "4e64e04364758c005f7b051f537f956c"
server_key_version: "encryptionkey@0"
I20260929 01:12:18.964640 25585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:18.979499 25585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:18.983904 25585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:18.984205 25585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:18.985265 25585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:18.986711 25585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:18.986812 25585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:18.986970 25585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:18.987044 25585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:19.018186 25446 heartbeater.cc:503] Master 127.5.198.254:37215 was elected leader, sending a full tablet report...
I20260929 01:12:19.020596 25585 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:39305
I20260929 01:12:19.020648 25712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:39305 every 8 connection(s)
I20260929 01:12:19.021914 25585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:12:19.030958 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 25585
I20260929 01:12:19.031165 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:12:19.031659 5915 external_mini_cluster.cc:1428] Setting key 698edb440b611e55fcac0e6d963a077f
Sep 29 01:12:19 dist-test-slave-41cl krb5kdc[25212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644338, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.195@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:12:19.042603 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.024367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:49081 (local address 127.5.198.254:37215)
0929 01:12:19.024833 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:19.024838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:19.025678 (+ 840us) server_negotiation.cc:407] Connection header received
0929 01:12:19.027746 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:19.027751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:19.027843 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:19.027958 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:19.029457 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.030255 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.031880 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.032385 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.032542 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.036445 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:19.036451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:19.036454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:19.036557 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:19.039489 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:19.040292 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:19.040298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:19.040301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:19.040442 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:19.040911 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:19.040915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:19.040917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:19.041214 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:19.041334 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:19.042168 (+ 834us) server_negotiation.cc:299] Negotiation successful
0929 01:12:19.042335 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":274,"threads_started":1}
I20260929 01:12:19.043694 25716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.024866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:37215 (local address 127.5.198.195:49081)
0929 01:12:19.025535 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:19.025573 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:19.027479 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:19.028188 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:19.028195 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:19.028768 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:19.029297 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:19.029313 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.030414 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.030418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:19.031733 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:19.031750 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.032647 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.032655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:19.032984 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.033764 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:12:19.033792 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:19.036228 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:19.039633 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:19.039639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:19.039651 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:19.040131 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:19.040612 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:19.040617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:19.040621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:19.040804 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:19.041360 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:19.041366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:19.041761 (+ 395us) client_negotiation.cc:769] Sending connection context
0929 01:12:19.042032 (+ 271us) client_negotiation.cc:240] Negotiation successful
0929 01:12:19.042400 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":253,"threads_started":1}
I20260929 01:12:19.045259 25713 heartbeater.cc:347] Connected to a master server at 127.5.198.254:37215
I20260929 01:12:19.045572 25713 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:19.046384 25713 heartbeater.cc:511] Master 127.5.198.254:37215 requested a full tablet report, sending...
I20260929 01:12:19.047756 25257 ts_manager.cc:194] Registered new tserver with Master: 46ed9cdf919941b39b668ad1401a7504 (127.5.198.195:39305)
I20260929 01:12:19.048745 25257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.198.195@KRBTEST.COM'} at 127.5.198.195:49081
I20260929 01:12:19.059624 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:12:19.076906 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.063895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.5.198.254:37215)
0929 01:12:19.064063 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:19.064068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:19.064458 (+ 390us) server_negotiation.cc:407] Connection header received
0929 01:12:19.064786 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:19.064790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:19.064880 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:19.064967 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:19.066445 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.067199 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.068689 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.069219 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.069325 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.071854 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:19.071858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:19.071862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:19.071894 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:19.074423 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:19.075176 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:19.075180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:19.075182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:19.075313 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:19.075644 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:19.075647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:19.075649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:19.075974 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:19.076168 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:19.076480 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 01:12:19.076681 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:12:19.080960 25257 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:12:19.087852 25257 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:12:19.125357 25730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.116257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.5.198.195:39305)
0929 01:12:19.117418 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:19.117430 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:19.118053 (+ 623us) server_negotiation.cc:407] Connection header received
0929 01:12:19.118292 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:19.118299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:19.118485 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:19.118671 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:19.119478 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.120878 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.122987 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.124168 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.124305 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.124572 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:19.124849 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 01:12:19.125004 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":760,"threads_started":1}
I20260929 01:12:19.125442 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.117366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:39305 (local address 127.0.0.1:47412)
0929 01:12:19.117923 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:19.117939 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:19.118121 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:19.118800 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:19.118806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:19.118836 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:19.119265 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:19.119276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.121022 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.121028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:19.122774 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:19.122791 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.124356 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.124363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:19.124595 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.124661 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 01:12:19.124809 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 01:12:19.124996 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":83,"spinlock_wait_cycles":896,"thread_start_us":294,"threads_started":1}
I20260929 01:12:19.126392 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.101030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:39837 (local address 127.0.0.1:60762)
0929 01:12:19.114513 (+ 13483us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:19.114585 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:19.114849 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:19.118544 (+ 3695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:19.118548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:19.118575 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:19.119021 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:19.119037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.120899 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.120904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:19.123361 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:19.123375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.125464 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.125471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:19.125668 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.125708 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 01:12:19.125794 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 01:12:19.125980 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13103,"mutex_wait_us":98,"thread_start_us":14598,"threads_started":1}
I20260929 01:12:19.126585 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.101350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:35945 (local address 127.0.0.1:37774)
0929 01:12:19.115424 (+ 14074us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:19.115442 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:19.115554 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:19.117905 (+ 2351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:19.117917 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:19.117947 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:19.118369 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:19.118385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.120362 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.120368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:19.123474 (+ 3106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:19.123484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.125697 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.125702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:19.125878 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.125907 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 01:12:19.126009 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 01:12:19.126138 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13962,"mutex_wait_us":22,"spinlock_wait_cycles":24832,"thread_start_us":13744,"threads_started":1}
I20260929 01:12:19.126839 25729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.101710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.5.198.193:35945)
0929 01:12:19.116540 (+ 14830us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:19.116554 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:19.116596 (+ 42us) server_negotiation.cc:407] Connection header received
0929 01:12:19.116737 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:19.116749 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:19.117063 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:19.117265 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:19.118581 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.120224 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.123642 (+ 3418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.125162 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.125280 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.125552 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:19.126061 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 01:12:19.126319 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14580,"thread_start_us":14937,"threads_started":1}
I20260929 01:12:19.127130 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.101241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.5.198.194:39837)
0929 01:12:19.117963 (+ 16722us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:19.117971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:19.117999 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:12:19.118100 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:19.118119 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:19.118341 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:19.118538 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:19.119228 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.120731 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.123561 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.124988 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.125122 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.125303 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:19.126630 (+ 1327us) server_negotiation.cc:299] Negotiation successful
0929 01:12:19.126801 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16568,"thread_start_us":16431,"threads_started":1}
I20260929 01:12:19.129640 25648 tablet_service.cc:1467] Processing CreateTablet for tablet 51f81c6ac94e4960ab1626e44e748f5f (DEFAULT_TABLE table=test-table [id=6b9654b7cc4d4e3fae0e1fb4e8877392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:19.130792 25648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f81c6ac94e4960ab1626e44e748f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:19.130293 25381 tablet_service.cc:1467] Processing CreateTablet for tablet 51f81c6ac94e4960ab1626e44e748f5f (DEFAULT_TABLE table=test-table [id=6b9654b7cc4d4e3fae0e1fb4e8877392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:19.130294 25515 tablet_service.cc:1467] Processing CreateTablet for tablet 51f81c6ac94e4960ab1626e44e748f5f (DEFAULT_TABLE table=test-table [id=6b9654b7cc4d4e3fae0e1fb4e8877392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:19.131582 25381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f81c6ac94e4960ab1626e44e748f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:19.131613 25515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f81c6ac94e4960ab1626e44e748f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:19.142573 25732 tablet_bootstrap.cc:492] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504: Bootstrap starting.
I20260929 01:12:19.146430 25734 tablet_bootstrap.cc:492] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef: Bootstrap starting.
I20260929 01:12:19.147487 25732 tablet_bootstrap.cc:654] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:19.145583 25733 tablet_bootstrap.cc:492] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d: Bootstrap starting.
I20260929 01:12:19.148845 25732 log.cc:824] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:19.149837 25734 tablet_bootstrap.cc:654] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:19.150982 25734 log.cc:824] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:19.151547 25733 tablet_bootstrap.cc:654] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:19.152011 25732 tablet_bootstrap.cc:492] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504: No bootstrap required, opened a new log
I20260929 01:12:19.152275 25732 ts_tablet_manager.cc:1397] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 01:12:19.152786 25733 log.cc:824] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:19.154052 25734 tablet_bootstrap.cc:492] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef: No bootstrap required, opened a new log
I20260929 01:12:19.154294 25734 ts_tablet_manager.cc:1397] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 01:12:19.155437 25733 tablet_bootstrap.cc:492] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d: No bootstrap required, opened a new log
I20260929 01:12:19.155675 25733 ts_tablet_manager.cc:1397] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260929 01:12:19.162575 25733 raft_consensus.cc:348] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ef551b67d543e8b26a7d04ab8d49ef" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39837 } } peers { permanent_uuid: "0e0503c99acd4c8089ddd3a141a4648d" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35945 } } peers { permanent_uuid: "46ed9cdf919941b39b668ad1401a7504" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39305 } }
I20260929 01:12:19.162921 25733 raft_consensus.cc:374] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:19.163028 25733 raft_consensus.cc:729] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e0503c99acd4c8089ddd3a141a4648d, State: Initialized, Role: FOLLOWER
I20260929 01:12:19.162792 25732 raft_consensus.cc:348] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ef551b67d543e8b26a7d04ab8d49ef" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39837 } } peers { permanent_uuid: "0e0503c99acd4c8089ddd3a141a4648d" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35945 } } peers { permanent_uuid: "46ed9cdf919941b39b668ad1401a7504" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39305 } }
I20260929 01:12:19.163242 25732 raft_consensus.cc:374] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:19.163334 25732 raft_consensus.cc:729] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46ed9cdf919941b39b668ad1401a7504, State: Initialized, Role: FOLLOWER
I20260929 01:12:19.163700 25733 consensus_queue.cc:260] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ef551b67d543e8b26a7d04ab8d49ef" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39837 } } peers { permanent_uuid: "0e0503c99acd4c8089ddd3a141a4648d" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35945 } } peers { permanent_uuid: "46ed9cdf919941b39b668ad1401a7504" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39305 } }
I20260929 01:12:19.163897 25732 consensus_queue.cc:260] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ef551b67d543e8b26a7d04ab8d49ef" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39837 } } peers { permanent_uuid: "0e0503c99acd4c8089ddd3a141a4648d" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35945 } } peers { permanent_uuid: "46ed9cdf919941b39b668ad1401a7504" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39305 } }
I20260929 01:12:19.164726 25734 raft_consensus.cc:348] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ef551b67d543e8b26a7d04ab8d49ef" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39837 } } peers { permanent_uuid: "0e0503c99acd4c8089ddd3a141a4648d" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35945 } } peers { permanent_uuid: "46ed9cdf919941b39b668ad1401a7504" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39305 } }
I20260929 01:12:19.165218 25734 raft_consensus.cc:374] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:19.165324 25734 raft_consensus.cc:729] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ef551b67d543e8b26a7d04ab8d49ef, State: Initialized, Role: FOLLOWER
I20260929 01:12:19.166118 25734 consensus_queue.cc:260] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ef551b67d543e8b26a7d04ab8d49ef" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39837 } } peers { permanent_uuid: "0e0503c99acd4c8089ddd3a141a4648d" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35945 } } peers { permanent_uuid: "46ed9cdf919941b39b668ad1401a7504" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39305 } }
I20260929 01:12:19.166960 25733 ts_tablet_manager.cc:1428] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d: Time spent starting tablet: real 0.011s user 0.005s sys 0.005s
I20260929 01:12:19.171211 25713 heartbeater.cc:503] Master 127.5.198.254:37215 was elected leader, sending a full tablet report...
I20260929 01:12:19.172340 25732 ts_tablet_manager.cc:1428] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504: Time spent starting tablet: real 0.020s user 0.013s sys 0.008s
I20260929 01:12:19.173207 25580 heartbeater.cc:503] Master 127.5.198.254:37215 was elected leader, sending a full tablet report...
I20260929 01:12:19.174070 25734 ts_tablet_manager.cc:1428] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s
I20260929 01:12:19.230347 25740 raft_consensus.cc:482] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:12:19.230762 25740 raft_consensus.cc:504] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ef551b67d543e8b26a7d04ab8d49ef" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39837 } } peers { permanent_uuid: "0e0503c99acd4c8089ddd3a141a4648d" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35945 } } peers { permanent_uuid: "46ed9cdf919941b39b668ad1401a7504" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39305 } }
I20260929 01:12:19.232607 25740 leader_election.cc:288] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e0503c99acd4c8089ddd3a141a4648d (127.5.198.193:35945), 46ed9cdf919941b39b668ad1401a7504 (127.5.198.195:39305)
W20260929 01:12:19.236405 25447 tablet.cc:2369] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:12:19.239900 25729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.233029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:47303 (local address 127.5.198.193:35945)
0929 01:12:19.233244 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:19.233248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:19.233662 (+ 414us) server_negotiation.cc:407] Connection header received
0929 01:12:19.233925 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:19.233931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:19.234015 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:19.234100 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:19.234774 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.235487 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.237943 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.238941 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.239123 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.239211 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:19.239536 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 01:12:19.239660 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:12:19.239928 25730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.233941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:57511 (local address 127.5.198.195:39305)
0929 01:12:19.234198 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:19.234205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:19.234317 (+ 112us) server_negotiation.cc:407] Connection header received
0929 01:12:19.234532 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:19.234543 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:19.234612 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:19.234708 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:19.235208 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.235923 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.238058 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.238963 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.239158 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.239236 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:19.239553 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 01:12:19.239672 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 01:12:19.240084 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.232855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:35945 (local address 127.5.198.194:47303)
0929 01:12:19.233529 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:19.233550 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:19.233701 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:19.234304 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:19.234307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:19.234328 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:19.234633 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:19.234640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.235621 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.235629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:19.237755 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:19.237767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.239172 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.239179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:19.239364 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.239395 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 01:12:19.239503 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 01:12:19.239639 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":60,"thread_start_us":339,"threads_started":1}
I20260929 01:12:19.240124 25742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.233624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:39305 (local address 127.5.198.194:57511)
0929 01:12:19.234195 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:19.234214 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:19.234390 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:19.234720 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:19.234725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:19.234748 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:19.235060 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:19.235070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.236059 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.236070 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:19.237901 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:19.237917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.239205 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.239211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:19.239397 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.239420 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 01:12:19.239527 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 01:12:19.239660 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":26,"thread_start_us":267,"threads_started":1}
I20260929 01:12:19.241216 25668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f81c6ac94e4960ab1626e44e748f5f" candidate_uuid: "99ef551b67d543e8b26a7d04ab8d49ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46ed9cdf919941b39b668ad1401a7504" is_pre_election: true
I20260929 01:12:19.241220 25401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f81c6ac94e4960ab1626e44e748f5f" candidate_uuid: "99ef551b67d543e8b26a7d04ab8d49ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e0503c99acd4c8089ddd3a141a4648d" is_pre_election: true
I20260929 01:12:19.241688 25401 raft_consensus.cc:2454] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ef551b67d543e8b26a7d04ab8d49ef in term 0.
I20260929 01:12:19.241688 25668 raft_consensus.cc:2454] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99ef551b67d543e8b26a7d04ab8d49ef in term 0.
I20260929 01:12:19.242321 25469 leader_election.cc:302] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e0503c99acd4c8089ddd3a141a4648d, 99ef551b67d543e8b26a7d04ab8d49ef; no voters:
I20260929 01:12:19.242683 25740 raft_consensus.cc:2781] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:12:19.242835 25740 raft_consensus.cc:482] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:12:19.242946 25740 raft_consensus.cc:3037] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:19.245291 25740 raft_consensus.cc:504] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ef551b67d543e8b26a7d04ab8d49ef" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39837 } } peers { permanent_uuid: "0e0503c99acd4c8089ddd3a141a4648d" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35945 } } peers { permanent_uuid: "46ed9cdf919941b39b668ad1401a7504" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39305 } }
I20260929 01:12:19.245905 25740 leader_election.cc:288] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [CANDIDATE]: Term 1 election: Requested vote from peers 0e0503c99acd4c8089ddd3a141a4648d (127.5.198.193:35945), 46ed9cdf919941b39b668ad1401a7504 (127.5.198.195:39305)
I20260929 01:12:19.246264 25401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f81c6ac94e4960ab1626e44e748f5f" candidate_uuid: "99ef551b67d543e8b26a7d04ab8d49ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e0503c99acd4c8089ddd3a141a4648d"
I20260929 01:12:19.246333 25668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f81c6ac94e4960ab1626e44e748f5f" candidate_uuid: "99ef551b67d543e8b26a7d04ab8d49ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46ed9cdf919941b39b668ad1401a7504"
I20260929 01:12:19.246459 25401 raft_consensus.cc:3037] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:19.246577 25668 raft_consensus.cc:3037] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:19.248731 25401 raft_consensus.cc:2454] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ef551b67d543e8b26a7d04ab8d49ef in term 1.
W20260929 01:12:19.248824 25581 tablet.cc:2369] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:12:19.248884 25668 raft_consensus.cc:2454] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99ef551b67d543e8b26a7d04ab8d49ef in term 1.
I20260929 01:12:19.249140 25469 leader_election.cc:302] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e0503c99acd4c8089ddd3a141a4648d, 99ef551b67d543e8b26a7d04ab8d49ef; no voters:
I20260929 01:12:19.249454 25740 raft_consensus.cc:2781] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:19.249894 25740 raft_consensus.cc:686] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [term 1 LEADER]: Becoming Leader. State: Replica: 99ef551b67d543e8b26a7d04ab8d49ef, State: Running, Role: LEADER
I20260929 01:12:19.250324 25740 consensus_queue.cc:237] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ef551b67d543e8b26a7d04ab8d49ef" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39837 } } peers { permanent_uuid: "0e0503c99acd4c8089ddd3a141a4648d" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35945 } } peers { permanent_uuid: "46ed9cdf919941b39b668ad1401a7504" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39305 } }
I20260929 01:12:19.254848 25256 catalog_manager.cc:5495] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 99ef551b67d543e8b26a7d04ab8d49ef (127.5.198.194). New cstate: current_term: 1 leader_uuid: "99ef551b67d543e8b26a7d04ab8d49ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99ef551b67d543e8b26a7d04ab8d49ef" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 39837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e0503c99acd4c8089ddd3a141a4648d" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46ed9cdf919941b39b668ad1401a7504" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39305 } health_report { overall_health: UNKNOWN } } }
W20260929 01:12:19.273494 25714 tablet.cc:2369] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:12:19.279754 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.274504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.5.198.194:39837)
0929 01:12:19.274687 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:19.274692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:19.274851 (+ 159us) server_negotiation.cc:407] Connection header received
0929 01:12:19.275008 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:19.275012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:19.275087 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:19.275218 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:12:19.275536 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.276233 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.277735 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.278146 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.278256 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.278766 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:12:19.279244 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:12:19.279362 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:19.279487 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 01:12:19.279548 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:12:19.287801 25401 raft_consensus.cc:1261] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d [term 1 FOLLOWER]: Refusing update from remote peer 99ef551b67d543e8b26a7d04ab8d49ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:12:19.287853 25668 raft_consensus.cc:1261] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504 [term 1 FOLLOWER]: Refusing update from remote peer 99ef551b67d543e8b26a7d04ab8d49ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:12:19.288834 25743 consensus_queue.cc:1035] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "46ed9cdf919941b39b668ad1401a7504" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 39305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:12:19.289216 25740 consensus_queue.cc:1035] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e0503c99acd4c8089ddd3a141a4648d" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 35945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:12:19.300918 25746 mvcc.cc:205] Tried to move back new op lower bound from 7334479213712846848 to 7334479213572624384. Current Snapshot: MvccSnapshot[applied={T|T < 7334479213712846848}]
I20260929 01:12:19.319245 25256 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:12:19.319653 25256 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:12:19.325683 25256 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30f9082108174ac480495aecb0b37518: Sending DeleteTablet for 3 replicas of tablet 51f81c6ac94e4960ab1626e44e748f5f
I20260929 01:12:19.327427 25515 tablet_service.cc:1514] Processing DeleteTablet for tablet 51f81c6ac94e4960ab1626e44e748f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:19 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:60762
I20260929 01:12:19.329241 25381 tablet_service.cc:1514] Processing DeleteTablet for tablet 51f81c6ac94e4960ab1626e44e748f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:19 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:37774
I20260929 01:12:19.337651 25648 tablet_service.cc:1514] Processing DeleteTablet for tablet 51f81c6ac94e4960ab1626e44e748f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:19 UTC) from {username='kudu', principal='kudu/127.5.198.254@KRBTEST.COM'} at 127.0.0.1:47412
I20260929 01:12:19.341163 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 25318
I20260929 01:12:19.365201 25757 tablet_replica.cc:330] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d: stopping tablet replica
I20260929 01:12:19.365927 25758 tablet_replica.cc:330] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504: stopping tablet replica
I20260929 01:12:19.367062 25756 tablet_replica.cc:330] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef: stopping tablet replica
I20260929 01:12:19.367553 25756 raft_consensus.cc:2229] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:12:19.368239 25756 raft_consensus.cc:2258] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:19.371698 25756 ts_tablet_manager.cc:1905] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:19.373494 25757 raft_consensus.cc:2229] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:12:19.374389 25757 raft_consensus.cc:2258] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:19.380287 25757 ts_tablet_manager.cc:1905] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:19.380688 25756 ts_tablet_manager.cc:1918] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:12:19.380868 25756 log.cc:1196] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/51f81c6ac94e4960ab1626e44e748f5f
I20260929 01:12:19.381430 25756 ts_tablet_manager.cc:1939] T 51f81c6ac94e4960ab1626e44e748f5f P 99ef551b67d543e8b26a7d04ab8d49ef: Deleting consensus metadata
I20260929 01:12:19.383163 25244 catalog_manager.cc:4841] TS 99ef551b67d543e8b26a7d04ab8d49ef (127.5.198.194:39837): tablet 51f81c6ac94e4960ab1626e44e748f5f (table test-table [id=6b9654b7cc4d4e3fae0e1fb4e8877392]) successfully deleted
I20260929 01:12:19.384425 25758 raft_consensus.cc:2229] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:12:19.385272 25758 raft_consensus.cc:2258] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:19.388236 25758 ts_tablet_manager.cc:1905] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:19.389496 25757 ts_tablet_manager.cc:1918] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 29 01:12:19 dist-test-slave-41cl krb5kdc[25212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.193@KRBTEST.COM
I20260929 01:12:19.389667 25757 log.cc:1196] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-0/wal/wals/51f81c6ac94e4960ab1626e44e748f5f
I20260929 01:12:19.390209 25757 ts_tablet_manager.cc:1939] T 51f81c6ac94e4960ab1626e44e748f5f P 0e0503c99acd4c8089ddd3a141a4648d: Deleting consensus metadata
I20260929 01:12:19.392362 25242 catalog_manager.cc:4841] TS 0e0503c99acd4c8089ddd3a141a4648d (127.5.198.193:35945): tablet 51f81c6ac94e4960ab1626e44e748f5f (table test-table [id=6b9654b7cc4d4e3fae0e1fb4e8877392]) successfully deleted
I20260929 01:12:19.395522 25758 ts_tablet_manager.cc:1918] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:12:19.395738 25758 log.cc:1196] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/51f81c6ac94e4960ab1626e44e748f5f
I20260929 01:12:19.396389 25758 ts_tablet_manager.cc:1939] T 51f81c6ac94e4960ab1626e44e748f5f P 46ed9cdf919941b39b668ad1401a7504: Deleting consensus metadata
I20260929 01:12:19.398259 25242 catalog_manager.cc:4841] TS 46ed9cdf919941b39b668ad1401a7504 (127.5.198.195:39305): tablet 51f81c6ac94e4960ab1626e44e748f5f (table test-table [id=6b9654b7cc4d4e3fae0e1fb4e8877392]) successfully deleted
I20260929 01:12:19.398766 25729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.342803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37782 (local address 127.5.198.193:35945)
0929 01:12:19.342996 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:19.343002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:19.354972 (+ 11970us) server_negotiation.cc:407] Connection header received
0929 01:12:19.355479 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:19.355487 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:19.355594 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:19.355690 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:19.375393 (+ 19703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.376746 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.379018 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.379720 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.379845 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.390886 (+ 11041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:19.390903 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:19.390924 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:19.390991 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:19.394881 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:19.395855 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:19.395867 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:19.395872 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:19.396045 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:19.396702 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:19.396710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:19.396716 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:19.397174 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:19.397454 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:19.398069 (+ 615us) server_negotiation.cc:299] Negotiation successful
0929 01:12:19.398345 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:12:19.399926 25441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:19.490981 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 25318
I20260929 01:12:19.510183 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 25452
Sep 29 01:12:19 dist-test-slave-41cl krb5kdc[25212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.194@KRBTEST.COM
I20260929 01:12:19.525933 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.510930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60780 (local address 127.5.198.194:39837)
0929 01:12:19.511205 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:19.511211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:19.511230 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:12:19.511339 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:19.511344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:19.511420 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:19.511542 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:19.512857 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.513690 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.514949 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.515397 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.515540 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.519966 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:19.519980 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:19.519994 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:19.520026 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:19.523090 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:19.523765 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:19.523771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:19.523773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:19.523895 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:19.524273 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:19.524281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:19.524284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:19.524690 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:19.525061 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:19.525281 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 01:12:19.525493 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260929 01:12:19.526690 25575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:19.646317 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 25452
I20260929 01:12:19.664899 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 25585
Sep 29 01:12:19 dist-test-slave-41cl krb5kdc[25212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.195@KRBTEST.COM
I20260929 01:12:19.680289 25730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:19.665726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.5.198.195:39305)
0929 01:12:19.665919 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:19.665925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:19.665947 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:12:19.666076 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:19.666085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:19.666146 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:19.666250 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:19.667524 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.668314 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.669621 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:19.670012 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:19.670145 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:19.674740 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:19.674756 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:19.674772 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:19.674809 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:19.677608 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:19.678305 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:19.678313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:19.678315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:19.678444 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:19.678816 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:19.678822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:19.678825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:19.679168 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:19.679494 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:19.679746 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 01:12:19.679965 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:12:19.681100 25708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:19.771642 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 25585
I20260929 01:12:19.788293 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 25225
I20260929 01:12:19.789433 25287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:19.902688 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 25225
2026-09-29T01:12:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4336 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T01:12:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:19Z Disabled control of system clock
I20260929 01:12:19.979701 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:41799
--webserver_interface=127.5.198.254
--webserver_port=0
--builtin_ntp_servers=127.5.198.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:41799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:12:20.321871 25767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:20.322218 25767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:20.322304 25767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:20.329839 25767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:20.329959 25767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:20.329996 25767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:20.330046 25767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:20.330089 25767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:20.339509 25767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:41799
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:41799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:20.340837 25767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:20.342561 25767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:20.355154 25772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:20.355094 25773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:20.355317 25775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:20.356184 25767 server_base.cc:1034] running on GCE node
I20260929 01:12:20.357079 25767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:20.358606 25767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:20.359855 25767 hybrid_clock.cc:648] HybridClock initialized: now 1790644340359810 us; error 46 us; skew 500 ppm
I20260929 01:12:20.360270 25767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:12:20.362831 25767 webserver.cc:460] Webserver started at http://127.5.198.254:39799/ using document root <none> and password file <none>
I20260929 01:12:20.363656 25767 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:20.363780 25767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:20.364135 25767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:20.367255 25767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "cefe4fff138342e3b7689749811844ff"
format_stamp: "Formatted at 2026-09-29 01:12:20 on dist-test-slave-41cl"
server_key: "42d965cf68be6c39ddb829ba88c383eb"
server_key_iv: "09d42cb50e8af0bf0e0086ecdbfeb71d"
server_key_version: "encryptionkey@0"
I20260929 01:12:20.368181 25767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "cefe4fff138342e3b7689749811844ff"
format_stamp: "Formatted at 2026-09-29 01:12:20 on dist-test-slave-41cl"
server_key: "42d965cf68be6c39ddb829ba88c383eb"
server_key_iv: "09d42cb50e8af0bf0e0086ecdbfeb71d"
server_key_version: "encryptionkey@0"
I20260929 01:12:20.373680 25767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260929 01:12:20.377267 25781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:20.378687 25767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:12:20.378876 25767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "cefe4fff138342e3b7689749811844ff"
format_stamp: "Formatted at 2026-09-29 01:12:20 on dist-test-slave-41cl"
server_key: "42d965cf68be6c39ddb829ba88c383eb"
server_key_iv: "09d42cb50e8af0bf0e0086ecdbfeb71d"
server_key_version: "encryptionkey@0"
I20260929 01:12:20.379102 25767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:20.405287 25767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:20.406152 25767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:20.406440 25767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:20.425447 25767 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:41799
I20260929 01:12:20.425488 25832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:41799 every 8 connection(s)
I20260929 01:12:20.426865 25767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:20.430677 25833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:20.434762 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 25767
I20260929 01:12:20.434975 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:20.435459 5915 external_mini_cluster.cc:1428] Setting key 68f34fe542944613f7920390a2e9a9c1
I20260929 01:12:20.443843 25833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff: Bootstrap starting.
I20260929 01:12:20.447422 25833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:20.448589 25833 log.cc:824] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:20.449839 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:20.437397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36504 (local address 127.5.198.254:41799)
0929 01:12:20.438693 (+ 1296us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:20.438704 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:20.438746 (+ 42us) server_negotiation.cc:407] Connection header received
0929 01:12:20.440379 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:20.440401 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:20.440970 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:20.441414 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:20.442018 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:20.443258 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:20.444780 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:20.445487 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:20.445702 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:20.447104 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:20.447114 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:20.447128 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:20.447756 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:20.447854 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:20.447878 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:20.448011 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:20.448234 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 01:12:20.448620 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":569,"threads_started":1}
I20260929 01:12:20.451938 25833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff: No bootstrap required, opened a new log
I20260929 01:12:20.459674 25833 raft_consensus.cc:348] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefe4fff138342e3b7689749811844ff" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 41799 } }
I20260929 01:12:20.460062 25833 raft_consensus.cc:374] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:20.460175 25833 raft_consensus.cc:729] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cefe4fff138342e3b7689749811844ff, State: Initialized, Role: FOLLOWER
I20260929 01:12:20.460845 25833 consensus_queue.cc:260] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefe4fff138342e3b7689749811844ff" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 41799 } }
I20260929 01:12:20.461097 25833 raft_consensus.cc:388] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:20.461246 25833 raft_consensus.cc:482] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:20.461452 25833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:20.463869 25833 raft_consensus.cc:504] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefe4fff138342e3b7689749811844ff" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 41799 } }
I20260929 01:12:20.464311 25833 leader_election.cc:302] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cefe4fff138342e3b7689749811844ff; no voters:
I20260929 01:12:20.464932 25833 leader_election.cc:288] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:20.465158 25838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:20.465865 25838 raft_consensus.cc:686] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [term 1 LEADER]: Becoming Leader. State: Replica: cefe4fff138342e3b7689749811844ff, State: Running, Role: LEADER
I20260929 01:12:20.466372 25838 consensus_queue.cc:237] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefe4fff138342e3b7689749811844ff" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 41799 } }
I20260929 01:12:20.467141 25833 sys_catalog.cc:564] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:20.469764 25840 sys_catalog.cc:455] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader cefe4fff138342e3b7689749811844ff. Latest consensus state: current_term: 1 leader_uuid: "cefe4fff138342e3b7689749811844ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefe4fff138342e3b7689749811844ff" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 41799 } } }
I20260929 01:12:20.470278 25840 sys_catalog.cc:458] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:20.469729 25839 sys_catalog.cc:455] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cefe4fff138342e3b7689749811844ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefe4fff138342e3b7689749811844ff" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 41799 } } }
I20260929 01:12:20.470817 25839 sys_catalog.cc:458] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:20.471758 25845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:20.476188 25845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:20.485970 25845 catalog_manager.cc:1347] Generated new cluster ID: 419ba08a9cba46939529cbd108a6443c
I20260929 01:12:20.486102 25845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:20.494109 25845 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:20.495898 25845 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:20.511392 25845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff: Generated new TSK 0
I20260929 01:12:20.512449 25845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:20.523341 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.193:0
--local_ip_for_outbound_sockets=127.5.198.193
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:41799
--builtin_ntp_servers=127.5.198.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:12:20.877161 25857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:20.877426 25857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:20.877581 25857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:20.885182 25857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:20.885301 25857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:20.885469 25857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.193
I20260929 01:12:20.895246 25857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.198.193
--webserver_port=0
--tserver_master_addrs=127.5.198.254:41799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.193
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:20.896739 25857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:20.898643 25857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:20.912352 25862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:20.912523 25865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:20.912531 25863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:20.913652 25857 server_base.cc:1034] running on GCE node
I20260929 01:12:20.914240 25857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:20.915158 25857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:20.916366 25857 hybrid_clock.cc:648] HybridClock initialized: now 1790644340916303 us; error 65 us; skew 500 ppm
I20260929 01:12:20.916800 25857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:12:20.919303 25857 webserver.cc:460] Webserver started at http://127.5.198.193:37887/ using document root <none> and password file <none>
I20260929 01:12:20.920023 25857 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:20.920111 25857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:20.920498 25857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:20.923331 25857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/data/instance:
uuid: "6511dd4f4c864f088e3eaf6a6c48000a"
format_stamp: "Formatted at 2026-09-29 01:12:20 on dist-test-slave-41cl"
server_key: "041e6d017ebefc71de0ef0b6b2057d95"
server_key_iv: "8ad04a91c73c532cafee866e64bb5e20"
server_key_version: "encryptionkey@0"
I20260929 01:12:20.924139 25857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance:
uuid: "6511dd4f4c864f088e3eaf6a6c48000a"
format_stamp: "Formatted at 2026-09-29 01:12:20 on dist-test-slave-41cl"
server_key: "041e6d017ebefc71de0ef0b6b2057d95"
server_key_iv: "8ad04a91c73c532cafee866e64bb5e20"
server_key_version: "encryptionkey@0"
I20260929 01:12:20.929291 25857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 01:12:20.933063 25871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:20.934387 25857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:12:20.934594 25857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/wal
uuid: "6511dd4f4c864f088e3eaf6a6c48000a"
format_stamp: "Formatted at 2026-09-29 01:12:20 on dist-test-slave-41cl"
server_key: "041e6d017ebefc71de0ef0b6b2057d95"
server_key_iv: "8ad04a91c73c532cafee866e64bb5e20"
server_key_version: "encryptionkey@0"
I20260929 01:12:20.934840 25857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:20.959384 25857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:20.960319 25857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:20.960727 25857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:20.961794 25857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:20.963389 25857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:20.963558 25857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:20.963690 25857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:20.963778 25857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:21.000160 25857 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.193:39129
I20260929 01:12:21.000257 25983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.193:39129 every 8 connection(s)
I20260929 01:12:21.001601 25857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/data/info.pb
I20260929 01:12:21.008762 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 25857
I20260929 01:12:21.008942 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-0/wal/instance
I20260929 01:12:21.009447 5915 external_mini_cluster.cc:1428] Setting key 2e34472b5494d65bf424da9c982f57bf
I20260929 01:12:21.012516 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.194:0
--local_ip_for_outbound_sockets=127.5.198.194
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:41799
--builtin_ntp_servers=127.5.198.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:12:21.015527 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:21.004376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.193:57111 (local address 127.5.198.254:41799)
0929 01:12:21.004865 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:21.004870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:21.006033 (+ 1163us) server_negotiation.cc:407] Connection header received
0929 01:12:21.007889 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:21.007894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:21.007984 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:21.008112 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:21.009341 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.010231 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.012357 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.012940 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.013035 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:21.014787 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:21.014792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:21.014795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:21.014813 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:21.014848 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:21.014860 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:21.014930 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:21.015193 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 01:12:21.015269 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":253,"threads_started":1}
I20260929 01:12:21.017055 25987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:21.004978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:41799 (local address 127.5.198.193:57111)
0929 01:12:21.005882 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:21.005922 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:21.007628 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:21.008351 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:21.008359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:21.008447 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:21.009152 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:21.009171 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.010424 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.010429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:21.012207 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:21.012225 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.013167 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.013177 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:21.013525 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:21.014415 (+ 890us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:12:21.014421 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:21.014475 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:12:21.014584 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:21.014961 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:21.014975 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:21.014986 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 01:12:21.015175 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 01:12:21.015606 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":284,"threads_started":1}
I20260929 01:12:21.018630 25984 heartbeater.cc:347] Connected to a master server at 127.5.198.254:41799
I20260929 01:12:21.018935 25984 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:21.019707 25984 heartbeater.cc:511] Master 127.5.198.254:41799 requested a full tablet report, sending...
I20260929 01:12:21.022019 25798 ts_manager.cc:194] Registered new tserver with Master: 6511dd4f4c864f088e3eaf6a6c48000a (127.5.198.193:39129)
I20260929 01:12:21.023509 25798 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.198.193:57111
W20260929 01:12:21.343014 25988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:21.343287 25988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:21.343426 25988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:21.350728 25988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:21.350828 25988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:21.351004 25988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.194
I20260929 01:12:21.360262 25988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.198.194
--webserver_port=0
--tserver_master_addrs=127.5.198.254:41799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.194
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:21.361598 25988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:21.363300 25988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:21.376735 25993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:21.376763 25996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:21.376840 25994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:21.377599 25988 server_base.cc:1034] running on GCE node
I20260929 01:12:21.378239 25988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:21.379274 25988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:21.380561 25988 hybrid_clock.cc:648] HybridClock initialized: now 1790644341380501 us; error 59 us; skew 500 ppm
I20260929 01:12:21.380903 25988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:12:21.383369 25988 webserver.cc:460] Webserver started at http://127.5.198.194:45741/ using document root <none> and password file <none>
I20260929 01:12:21.384073 25988 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:21.384199 25988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:21.384588 25988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:21.387389 25988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/data/instance:
uuid: "63c2fdf240044c9a8528f153a72a7a71"
format_stamp: "Formatted at 2026-09-29 01:12:21 on dist-test-slave-41cl"
server_key: "e26a324447d0ca3921f02561916ef42a"
server_key_iv: "604c41c120c911f674093e4b380a7305"
server_key_version: "encryptionkey@0"
I20260929 01:12:21.388175 25988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance:
uuid: "63c2fdf240044c9a8528f153a72a7a71"
format_stamp: "Formatted at 2026-09-29 01:12:21 on dist-test-slave-41cl"
server_key: "e26a324447d0ca3921f02561916ef42a"
server_key_iv: "604c41c120c911f674093e4b380a7305"
server_key_version: "encryptionkey@0"
I20260929 01:12:21.393327 25988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260929 01:12:21.396943 26002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:21.398260 25988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:12:21.398433 25988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/wal
uuid: "63c2fdf240044c9a8528f153a72a7a71"
format_stamp: "Formatted at 2026-09-29 01:12:21 on dist-test-slave-41cl"
server_key: "e26a324447d0ca3921f02561916ef42a"
server_key_iv: "604c41c120c911f674093e4b380a7305"
server_key_version: "encryptionkey@0"
I20260929 01:12:21.398679 25988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:21.431205 25988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:21.432039 25988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:21.432420 25988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:21.433476 25988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:21.434962 25988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:21.435093 25988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:21.435215 25988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:21.435299 25988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:21.469465 25988 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.194:44005
I20260929 01:12:21.469552 26114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.194:44005 every 8 connection(s)
I20260929 01:12:21.470777 25988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/data/info.pb
I20260929 01:12:21.471308 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 25988
I20260929 01:12:21.471515 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/wal/instance
I20260929 01:12:21.472137 5915 external_mini_cluster.cc:1428] Setting key c840186e6dfae0130bda0f4bbb44de00
I20260929 01:12:21.475296 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.198.195:0
--local_ip_for_outbound_sockets=127.5.198.195
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:41799
--builtin_ntp_servers=127.5.198.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:12:21.483891 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:21.473744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.194:46743 (local address 127.5.198.254:41799)
0929 01:12:21.473912 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:21.473916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:21.475562 (+ 1646us) server_negotiation.cc:407] Connection header received
0929 01:12:21.477429 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:21.477435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:21.477521 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:21.477640 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:21.478705 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.479440 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.481028 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.481429 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.481563 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:21.482972 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:21.482980 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:21.482985 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:21.483011 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:21.483054 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:21.483072 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:21.483177 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:21.483474 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 01:12:21.483590 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:12:21.485265 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:21.474405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:41799 (local address 127.5.198.194:46743)
0929 01:12:21.475422 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:21.475465 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:21.477182 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:21.477869 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:21.477877 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:21.477983 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:21.478536 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:21.478551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.479597 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.479602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:21.480881 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:21.480893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.481651 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.481659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:21.481877 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:21.482652 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:12:21.482655 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:21.482697 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:12:21.482770 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:21.483184 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:21.483192 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:21.483199 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 01:12:21.483435 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 01:12:21.483837 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":332,"threads_started":1}
I20260929 01:12:21.486634 26115 heartbeater.cc:347] Connected to a master server at 127.5.198.254:41799
I20260929 01:12:21.486918 26115 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:21.487670 26115 heartbeater.cc:511] Master 127.5.198.254:41799 requested a full tablet report, sending...
I20260929 01:12:21.489027 25798 ts_manager.cc:194] Registered new tserver with Master: 63c2fdf240044c9a8528f153a72a7a71 (127.5.198.194:44005)
I20260929 01:12:21.489854 25798 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.198.194:46743
W20260929 01:12:21.806491 26118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:21.806792 26118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:21.806954 26118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:21.814097 26118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:21.814224 26118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:21.814399 26118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.198.195
I20260929 01:12:21.823419 26118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.198.195
--webserver_port=0
--tserver_master_addrs=127.5.198.254:41799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.198.195
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:21.824770 26118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:21.826532 26118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:21.839732 26123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:21.839742 26124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:21.839776 26126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:21.840716 26118 server_base.cc:1034] running on GCE node
I20260929 01:12:21.841307 26118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:21.842226 26118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:21.843461 26118 hybrid_clock.cc:648] HybridClock initialized: now 1790644341843412 us; error 65 us; skew 500 ppm
I20260929 01:12:21.843776 26118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:12:21.846078 26118 webserver.cc:460] Webserver started at http://127.5.198.195:45027/ using document root <none> and password file <none>
I20260929 01:12:21.846747 26118 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:21.846879 26118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:21.847193 26118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:21.850016 26118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/data/instance:
uuid: "3828783d5d514b23abca314ddd11aacc"
format_stamp: "Formatted at 2026-09-29 01:12:21 on dist-test-slave-41cl"
server_key: "bf061a179f4c901022d814896cefec78"
server_key_iv: "002baa09dd1c1e17b210c03f6e38e950"
server_key_version: "encryptionkey@0"
I20260929 01:12:21.850807 26118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance:
uuid: "3828783d5d514b23abca314ddd11aacc"
format_stamp: "Formatted at 2026-09-29 01:12:21 on dist-test-slave-41cl"
server_key: "bf061a179f4c901022d814896cefec78"
server_key_iv: "002baa09dd1c1e17b210c03f6e38e950"
server_key_version: "encryptionkey@0"
I20260929 01:12:21.856186 26118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:12:21.861196 26132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:21.862952 26118 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260929 01:12:21.863125 26118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/wal
uuid: "3828783d5d514b23abca314ddd11aacc"
format_stamp: "Formatted at 2026-09-29 01:12:21 on dist-test-slave-41cl"
server_key: "bf061a179f4c901022d814896cefec78"
server_key_iv: "002baa09dd1c1e17b210c03f6e38e950"
server_key_version: "encryptionkey@0"
I20260929 01:12:21.863328 26118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:21.887316 26118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:21.888176 26118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:21.888585 26118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:21.889633 26118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:12:21.891232 26118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:12:21.891352 26118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:21.891489 26118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:12:21.891589 26118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:21.924793 26118 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.195:36101
I20260929 01:12:21.924878 26244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.195:36101 every 8 connection(s)
I20260929 01:12:21.926040 26118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/data/info.pb
I20260929 01:12:21.931933 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 26118
I20260929 01:12:21.932121 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/wal/instance
I20260929 01:12:21.932708 5915 external_mini_cluster.cc:1428] Setting key 952c303db566ba3a08f23ea346c5c652
I20260929 01:12:21.939168 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:21.928646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:38593 (local address 127.5.198.254:41799)
0929 01:12:21.928832 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:21.928836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:21.930162 (+ 1326us) server_negotiation.cc:407] Connection header received
0929 01:12:21.932109 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:21.932116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:21.932230 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:21.932331 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:21.933652 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.934670 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.936408 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.936890 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.936990 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:21.938425 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:21.938430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:21.938434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:21.938451 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:21.938484 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:21.938496 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:21.938573 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:21.938808 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 01:12:21.938931 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 01:12:21.940354 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:21.929142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:41799 (local address 127.5.198.195:38593)
0929 01:12:21.929985 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:21.930019 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:21.931760 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:21.932612 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:21.932623 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:21.932741 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:21.933448 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:21.933472 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.934825 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.934828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:21.936269 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:21.936280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.937147 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.937152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:21.937347 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:21.938051 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:12:21.938053 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:21.938092 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:12:21.938227 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:21.938610 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:21.938619 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:21.938626 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 01:12:21.938812 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 01:12:21.939181 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":259,"threads_started":1}
I20260929 01:12:21.941733 26245 heartbeater.cc:347] Connected to a master server at 127.5.198.254:41799
I20260929 01:12:21.942052 26245 heartbeater.cc:464] Registering TS with master...
I20260929 01:12:21.942775 26245 heartbeater.cc:511] Master 127.5.198.254:41799 requested a full tablet report, sending...
I20260929 01:12:21.944227 25798 ts_manager.cc:194] Registered new tserver with Master: 3828783d5d514b23abca314ddd11aacc (127.5.198.195:36101)
I20260929 01:12:21.945086 25798 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.198.195:38593
I20260929 01:12:21.947983 5915 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:12:21.957585 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:21.951860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36510 (local address 127.5.198.254:41799)
0929 01:12:21.952088 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:21.952097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:21.952526 (+ 429us) server_negotiation.cc:407] Connection header received
0929 01:12:21.952784 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:21.952790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:21.952871 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:21.952951 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:21.953374 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.954043 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.955350 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.955746 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.955835 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:21.956941 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:21.956945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:21.956948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:21.956965 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:21.956996 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:21.957008 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:21.957075 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:21.957204 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 01:12:21.957275 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:12:21.961256 25798 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:12:21.968123 25798 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:12:21.994571 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:21.980737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.5.198.195:36101)
0929 01:12:21.981605 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:21.981618 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:21.985868 (+ 4250us) server_negotiation.cc:407] Connection header received
0929 01:12:21.986527 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:21.986536 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:21.986796 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:21.986967 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:21.987864 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.989254 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.992208 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.993357 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.993519 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:21.993800 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:21.994065 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 01:12:21.994222 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":1133,"threads_started":1}
I20260929 01:12:21.994571 26257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:21.979677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.195:36101 (local address 127.0.0.1:48696)
0929 01:12:21.985722 (+ 6045us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:21.985763 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:21.986027 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:21.987360 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:21.987366 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:21.987397 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:21.987672 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:21.987680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.989406 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.989411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:21.991999 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:21.992013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.993568 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.993578 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:21.993801 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:21.993873 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 01:12:21.994004 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 01:12:21.994187 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5790,"thread_start_us":5642,"threads_started":1}
I20260929 01:12:21.997383 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:21.980056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39129 (local address 127.0.0.1:58788)
0929 01:12:21.988911 (+ 8855us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:21.988936 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:21.989148 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:21.990098 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:21.990103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:21.990127 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:21.990547 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:21.990560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.992367 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.992370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:21.994644 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:21.994659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.996630 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.996635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:21.996765 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:21.996787 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:12:21.996891 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 01:12:21.997013 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8736,"mutex_wait_us":78,"thread_start_us":8595,"threads_started":1}
I20260929 01:12:21.997440 26262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:21.981254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.5.198.193:39129)
0929 01:12:21.985131 (+ 3877us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:21.985145 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:21.989041 (+ 3896us) server_negotiation.cc:407] Connection header received
0929 01:12:21.989359 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:21.989369 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:21.989629 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:21.989791 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:21.990744 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.992230 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.994861 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.996074 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.996202 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:21.996445 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:21.996935 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 01:12:21.997098 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3654,"thread_start_us":3501,"threads_started":1}
I20260929 01:12:21.998678 26260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:21.979685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:44005 (local address 127.0.0.1:53526)
0929 01:12:21.989294 (+ 9609us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:21.989323 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:21.989471 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:21.990802 (+ 1331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:21.990806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:21.990831 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:21.991289 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:21.991303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.992857 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.992862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:21.994960 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:21.994973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.997924 (+ 2951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.997930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:21.998105 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:21.998135 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 01:12:21.998229 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 01:12:21.998348 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9462,"mutex_wait_us":513,"thread_start_us":8801,"threads_started":1}
I20260929 01:12:21.998869 26258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:21.980069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53526 (local address 127.5.198.194:44005)
0929 01:12:21.990033 (+ 9964us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:21.990039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:21.990062 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:12:21.990132 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:21.990136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:21.990334 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:21.990548 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:21.991492 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.992711 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.996065 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:21.997463 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:21.997604 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:21.997805 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:21.998303 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 01:12:21.998539 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9771,"thread_start_us":9671,"threads_started":1}
I20260929 01:12:21.999073 26180 tablet_service.cc:1467] Processing CreateTablet for tablet 53350943d4c9431d89418171fcba7836 (DEFAULT_TABLE table=test-table [id=d2b2b32c995d439fb7cf1476e7cee59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:22.000717 26180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53350943d4c9431d89418171fcba7836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:22.000946 25919 tablet_service.cc:1467] Processing CreateTablet for tablet 53350943d4c9431d89418171fcba7836 (DEFAULT_TABLE table=test-table [id=d2b2b32c995d439fb7cf1476e7cee59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:22.002296 25919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53350943d4c9431d89418171fcba7836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:22.002718 26050 tablet_service.cc:1467] Processing CreateTablet for tablet 53350943d4c9431d89418171fcba7836 (DEFAULT_TABLE table=test-table [id=d2b2b32c995d439fb7cf1476e7cee59c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:12:22.003937 26050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53350943d4c9431d89418171fcba7836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:22.015543 26263 tablet_bootstrap.cc:492] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc: Bootstrap starting.
I20260929 01:12:22.017819 26265 tablet_bootstrap.cc:492] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71: Bootstrap starting.
I20260929 01:12:22.019258 26264 tablet_bootstrap.cc:492] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a: Bootstrap starting.
I20260929 01:12:22.020313 26263 tablet_bootstrap.cc:654] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:22.021224 26265 tablet_bootstrap.cc:654] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:22.021590 26263 log.cc:824] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:22.022367 26265 log.cc:824] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:22.022789 26264 tablet_bootstrap.cc:654] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:22.024191 26264 log.cc:824] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:22.024271 26263 tablet_bootstrap.cc:492] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc: No bootstrap required, opened a new log
I20260929 01:12:22.024591 26263 ts_tablet_manager.cc:1397] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 01:12:22.026350 25984 heartbeater.cc:503] Master 127.5.198.254:41799 was elected leader, sending a full tablet report...
I20260929 01:12:22.027280 26264 tablet_bootstrap.cc:492] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a: No bootstrap required, opened a new log
I20260929 01:12:22.027530 26264 ts_tablet_manager.cc:1397] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 01:12:22.027657 26265 tablet_bootstrap.cc:492] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71: No bootstrap required, opened a new log
I20260929 01:12:22.027932 26265 ts_tablet_manager.cc:1397] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 01:12:22.035681 26264 raft_consensus.cc:348] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828783d5d514b23abca314ddd11aacc" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36101 } } peers { permanent_uuid: "63c2fdf240044c9a8528f153a72a7a71" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 44005 } } peers { permanent_uuid: "6511dd4f4c864f088e3eaf6a6c48000a" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39129 } }
I20260929 01:12:22.035707 26263 raft_consensus.cc:348] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828783d5d514b23abca314ddd11aacc" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36101 } } peers { permanent_uuid: "63c2fdf240044c9a8528f153a72a7a71" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 44005 } } peers { permanent_uuid: "6511dd4f4c864f088e3eaf6a6c48000a" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39129 } }
I20260929 01:12:22.036036 26264 raft_consensus.cc:374] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:22.036109 26263 raft_consensus.cc:374] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:22.036127 26264 raft_consensus.cc:729] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6511dd4f4c864f088e3eaf6a6c48000a, State: Initialized, Role: FOLLOWER
I20260929 01:12:22.036209 26263 raft_consensus.cc:729] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3828783d5d514b23abca314ddd11aacc, State: Initialized, Role: FOLLOWER
I20260929 01:12:22.036761 26264 consensus_queue.cc:260] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828783d5d514b23abca314ddd11aacc" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36101 } } peers { permanent_uuid: "63c2fdf240044c9a8528f153a72a7a71" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 44005 } } peers { permanent_uuid: "6511dd4f4c864f088e3eaf6a6c48000a" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39129 } }
I20260929 01:12:22.036892 26263 consensus_queue.cc:260] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828783d5d514b23abca314ddd11aacc" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36101 } } peers { permanent_uuid: "63c2fdf240044c9a8528f153a72a7a71" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 44005 } } peers { permanent_uuid: "6511dd4f4c864f088e3eaf6a6c48000a" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39129 } }
I20260929 01:12:22.038137 26245 heartbeater.cc:503] Master 127.5.198.254:41799 was elected leader, sending a full tablet report...
I20260929 01:12:22.038327 26265 raft_consensus.cc:348] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828783d5d514b23abca314ddd11aacc" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36101 } } peers { permanent_uuid: "63c2fdf240044c9a8528f153a72a7a71" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 44005 } } peers { permanent_uuid: "6511dd4f4c864f088e3eaf6a6c48000a" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39129 } }
I20260929 01:12:22.038722 26263 ts_tablet_manager.cc:1428] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260929 01:12:22.038733 26265 raft_consensus.cc:374] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:22.038818 26265 raft_consensus.cc:729] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c2fdf240044c9a8528f153a72a7a71, State: Initialized, Role: FOLLOWER
I20260929 01:12:22.039501 26265 consensus_queue.cc:260] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828783d5d514b23abca314ddd11aacc" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36101 } } peers { permanent_uuid: "63c2fdf240044c9a8528f153a72a7a71" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 44005 } } peers { permanent_uuid: "6511dd4f4c864f088e3eaf6a6c48000a" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39129 } }
I20260929 01:12:22.040509 26264 ts_tablet_manager.cc:1428] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260929 01:12:22.041347 26115 heartbeater.cc:503] Master 127.5.198.254:41799 was elected leader, sending a full tablet report...
I20260929 01:12:22.041950 26265 ts_tablet_manager.cc:1428] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260929 01:12:22.064544 26270 raft_consensus.cc:482] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:12:22.064832 26270 raft_consensus.cc:504] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828783d5d514b23abca314ddd11aacc" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36101 } } peers { permanent_uuid: "63c2fdf240044c9a8528f153a72a7a71" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 44005 } } peers { permanent_uuid: "6511dd4f4c864f088e3eaf6a6c48000a" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39129 } }
I20260929 01:12:22.066465 26270 leader_election.cc:288] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63c2fdf240044c9a8528f153a72a7a71 (127.5.198.194:44005), 6511dd4f4c864f088e3eaf6a6c48000a (127.5.198.193:39129)
I20260929 01:12:22.073271 26262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:22.067007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:42589 (local address 127.5.198.193:39129)
0929 01:12:22.067233 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:22.067239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:22.067260 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:12:22.067344 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:22.067348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:22.067417 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:22.067497 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:22.068103 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:22.069202 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:22.071253 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:22.072458 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:22.072616 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:22.072684 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:22.072974 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 01:12:22.073064 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 01:12:22.073388 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:22.066905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39129 (local address 127.5.198.195:42589)
0929 01:12:22.067072 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:22.067096 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:22.067216 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:22.067582 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:22.067589 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:22.067626 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:22.067908 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:22.067923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:22.069340 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:22.069346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:22.071095 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:22.071107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:22.072625 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:22.072631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:22.072805 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:22.072839 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 01:12:22.072952 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 01:12:22.073096 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":1}
I20260929 01:12:22.074541 26258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:22.067028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.198.195:58163 (local address 127.5.198.194:44005)
0929 01:12:22.067320 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:22.067327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:22.068286 (+ 959us) server_negotiation.cc:407] Connection header received
0929 01:12:22.068448 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:22.068452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:22.068596 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:22.068669 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:12:22.069250 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:22.070307 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:22.072436 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:22.073662 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:22.073809 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:22.073874 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:22.074165 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 01:12:22.074300 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260929 01:12:22.074584 26272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:22.066926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.194:44005 (local address 127.5.198.195:58163)
0929 01:12:22.068152 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:22.068204 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:22.068325 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:22.068717 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:22.068722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:22.068746 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:12:22.069063 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:22.069073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:22.070456 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:22.070459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:22.072255 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:22.072272 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:22.073809 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:22.073815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:22.073973 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:22.073994 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:12:22.074136 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 01:12:22.074255 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"mutex_wait_us":416,"thread_start_us":426,"threads_started":1}
I20260929 01:12:22.074405 25939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53350943d4c9431d89418171fcba7836" candidate_uuid: "3828783d5d514b23abca314ddd11aacc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6511dd4f4c864f088e3eaf6a6c48000a" is_pre_election: true
I20260929 01:12:22.074949 25939 raft_consensus.cc:2454] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3828783d5d514b23abca314ddd11aacc in term 0.
I20260929 01:12:22.075512 26070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53350943d4c9431d89418171fcba7836" candidate_uuid: "3828783d5d514b23abca314ddd11aacc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c2fdf240044c9a8528f153a72a7a71" is_pre_election: true
I20260929 01:12:22.075737 26135 leader_election.cc:302] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3828783d5d514b23abca314ddd11aacc, 6511dd4f4c864f088e3eaf6a6c48000a; no voters:
I20260929 01:12:22.076004 26070 raft_consensus.cc:2454] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3828783d5d514b23abca314ddd11aacc in term 0.
I20260929 01:12:22.076215 26270 raft_consensus.cc:2781] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:12:22.076371 26270 raft_consensus.cc:482] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:12:22.076532 26270 raft_consensus.cc:3037] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:22.078770 26270 raft_consensus.cc:504] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828783d5d514b23abca314ddd11aacc" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36101 } } peers { permanent_uuid: "63c2fdf240044c9a8528f153a72a7a71" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 44005 } } peers { permanent_uuid: "6511dd4f4c864f088e3eaf6a6c48000a" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39129 } }
I20260929 01:12:22.079470 26270 leader_election.cc:288] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [CANDIDATE]: Term 1 election: Requested vote from peers 63c2fdf240044c9a8528f153a72a7a71 (127.5.198.194:44005), 6511dd4f4c864f088e3eaf6a6c48000a (127.5.198.193:39129)
I20260929 01:12:22.079912 26070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53350943d4c9431d89418171fcba7836" candidate_uuid: "3828783d5d514b23abca314ddd11aacc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c2fdf240044c9a8528f153a72a7a71"
I20260929 01:12:22.079975 25939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53350943d4c9431d89418171fcba7836" candidate_uuid: "3828783d5d514b23abca314ddd11aacc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6511dd4f4c864f088e3eaf6a6c48000a"
I20260929 01:12:22.080124 26070 raft_consensus.cc:3037] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:22.080159 25939 raft_consensus.cc:3037] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:22.083442 26070 raft_consensus.cc:2454] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3828783d5d514b23abca314ddd11aacc in term 1.
I20260929 01:12:22.083436 25939 raft_consensus.cc:2454] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3828783d5d514b23abca314ddd11aacc in term 1.
I20260929 01:12:22.083832 26135 leader_election.cc:302] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3828783d5d514b23abca314ddd11aacc, 6511dd4f4c864f088e3eaf6a6c48000a; no voters:
I20260929 01:12:22.084106 26270 raft_consensus.cc:2781] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:22.084637 26270 raft_consensus.cc:686] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [term 1 LEADER]: Becoming Leader. State: Replica: 3828783d5d514b23abca314ddd11aacc, State: Running, Role: LEADER
I20260929 01:12:22.085105 26270 consensus_queue.cc:237] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828783d5d514b23abca314ddd11aacc" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36101 } } peers { permanent_uuid: "63c2fdf240044c9a8528f153a72a7a71" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 44005 } } peers { permanent_uuid: "6511dd4f4c864f088e3eaf6a6c48000a" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39129 } }
I20260929 01:12:22.090004 25796 catalog_manager.cc:5495] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3828783d5d514b23abca314ddd11aacc (127.5.198.195). New cstate: current_term: 1 leader_uuid: "3828783d5d514b23abca314ddd11aacc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3828783d5d514b23abca314ddd11aacc" member_type: VOTER last_known_addr { host: "127.5.198.195" port: 36101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c2fdf240044c9a8528f153a72a7a71" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 44005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6511dd4f4c864f088e3eaf6a6c48000a" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39129 } health_report { overall_health: UNKNOWN } } }
I20260929 01:12:22.115783 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:22.110013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.5.198.195:36101)
0929 01:12:22.110263 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:22.110268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:22.110290 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:12:22.110399 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:22.110404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:22.110479 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:22.110633 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:12:22.111065 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:22.111786 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:22.113230 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:22.113641 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:22.113737 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:22.114373 (+ 636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:12:22.115141 (+ 768us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:12:22.115304 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:22.115358 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 01:12:22.115454 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:12:22.124185 25939 raft_consensus.cc:1261] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a [term 1 FOLLOWER]: Refusing update from remote peer 3828783d5d514b23abca314ddd11aacc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:12:22.124186 26070 raft_consensus.cc:1261] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71 [term 1 FOLLOWER]: Refusing update from remote peer 3828783d5d514b23abca314ddd11aacc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:12:22.125193 26273 consensus_queue.cc:1035] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c2fdf240044c9a8528f153a72a7a71" member_type: VOTER last_known_addr { host: "127.5.198.194" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:12:22.125524 26270 consensus_queue.cc:1035] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6511dd4f4c864f088e3eaf6a6c48000a" member_type: VOTER last_known_addr { host: "127.5.198.193" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:12:22.137859 26276 mvcc.cc:205] Tried to move back new op lower bound from 7334479225330135040 to 7334479225184583680. Current Snapshot: MvccSnapshot[applied={T|T < 7334479225330135040}]
I20260929 01:12:22.138504 26277 mvcc.cc:205] Tried to move back new op lower bound from 7334479225330135040 to 7334479225184583680. Current Snapshot: MvccSnapshot[applied={T|T < 7334479225330135040}]
I20260929 01:12:22.151546 25796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:12:22.151830 25796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:12:22.155545 25796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cefe4fff138342e3b7689749811844ff: Sending DeleteTablet for 3 replicas of tablet 53350943d4c9431d89418171fcba7836
I20260929 01:12:22.156723 26180 tablet_service.cc:1514] Processing DeleteTablet for tablet 53350943d4c9431d89418171fcba7836 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:22 UTC) from {username='slave'} at 127.0.0.1:48696
I20260929 01:12:22.156876 26050 tablet_service.cc:1514] Processing DeleteTablet for tablet 53350943d4c9431d89418171fcba7836 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:22 UTC) from {username='slave'} at 127.0.0.1:53526
I20260929 01:12:22.156898 25919 tablet_service.cc:1514] Processing DeleteTablet for tablet 53350943d4c9431d89418171fcba7836 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:12:22 UTC) from {username='slave'} at 127.0.0.1:58788
I20260929 01:12:22.161325 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 25857
I20260929 01:12:22.168174 26287 tablet_replica.cc:330] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71: stopping tablet replica
I20260929 01:12:22.169487 26287 raft_consensus.cc:2229] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:12:22.170861 26287 raft_consensus.cc:2258] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:22.171808 26288 tablet_replica.cc:330] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a: stopping tablet replica
I20260929 01:12:22.172248 26288 raft_consensus.cc:2229] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:12:22.172866 26288 raft_consensus.cc:2258] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:22.175889 26287 ts_tablet_manager.cc:1905] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:22.175992 26286 tablet_replica.cc:330] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc: stopping tablet replica
I20260929 01:12:22.176373 26286 raft_consensus.cc:2229] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:12:22.177080 26286 raft_consensus.cc:2258] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:12:22.179167 26286 ts_tablet_manager.cc:1905] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:22.179155 26288 ts_tablet_manager.cc:1905] T 53350943d4c9431d89418171fcba7836 P 6511dd4f4c864f088e3eaf6a6c48000a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:12:22.184741 26262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:22.162536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.5.198.193:39129)
0929 01:12:22.162782 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:22.162791 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:22.175141 (+ 12350us) server_negotiation.cc:407] Connection header received
0929 01:12:22.175349 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:22.175355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:22.175459 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:22.175571 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:22.176333 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:22.177493 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:22.179586 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:22.180282 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:22.180388 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:22.181867 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:22.181877 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:22.181896 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:22.182745 (+ 849us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:22.182803 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:22.182828 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:22.182934 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:22.183435 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 01:12:22.183608 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 01:12:22.185724 25979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:22.187069 26286 ts_tablet_manager.cc:1918] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:12:22.187222 26286 log.cc:1196] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-2/wal/wals/53350943d4c9431d89418171fcba7836
I20260929 01:12:22.187723 26286 ts_tablet_manager.cc:1939] T 53350943d4c9431d89418171fcba7836 P 3828783d5d514b23abca314ddd11aacc: Deleting consensus metadata
I20260929 01:12:22.188606 26287 ts_tablet_manager.cc:1918] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:12:22.188777 26287 log.cc:1196] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71: Deleting WAL directory at /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790644023292677-5915-0/minicluster-data/ts-1/wal/wals/53350943d4c9431d89418171fcba7836
I20260929 01:12:22.189307 26287 ts_tablet_manager.cc:1939] T 53350943d4c9431d89418171fcba7836 P 63c2fdf240044c9a8528f153a72a7a71: Deleting consensus metadata
I20260929 01:12:22.189584 25785 catalog_manager.cc:4841] TS 3828783d5d514b23abca314ddd11aacc (127.5.198.195:36101): tablet 53350943d4c9431d89418171fcba7836 (table test-table [id=d2b2b32c995d439fb7cf1476e7cee59c]) successfully deleted
I20260929 01:12:22.190793 25783 catalog_manager.cc:4841] TS 63c2fdf240044c9a8528f153a72a7a71 (127.5.198.194:44005): tablet 53350943d4c9431d89418171fcba7836 (table test-table [id=d2b2b32c995d439fb7cf1476e7cee59c]) successfully deleted
I20260929 01:12:22.280056 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 25857
I20260929 01:12:22.304050 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 25988
W20260929 01:12:22.304080 25784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.198.193:39129 (error 108)
I20260929 01:12:22.307165 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:22.306653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39129 (local address 127.0.0.1:58810)
0929 01:12:22.306848 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:22.306974 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 01:12:22.307403 25784 catalog_manager.cc:4584] TS 6511dd4f4c864f088e3eaf6a6c48000a (127.5.198.193:39129): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 53350943d4c9431d89418171fcba7836: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39129: connect: Connection refused (error 111)
I20260929 01:12:22.318553 26258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:22.305075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53538 (local address 127.5.198.194:44005)
0929 01:12:22.305280 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:22.305287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:22.305672 (+ 385us) server_negotiation.cc:407] Connection header received
0929 01:12:22.305752 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:22.305758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:22.305892 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:22.306018 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:22.306420 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:22.307340 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:22.308809 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:22.309221 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:22.309365 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:22.317158 (+ 7793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:22.317168 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:22.317182 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:22.317766 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:22.317815 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:22.317835 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:22.317972 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:22.318140 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 01:12:22.318253 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 01:12:22.319301 26110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:22.361133 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:22.360592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39129 (local address 127.0.0.1:58826)
0929 01:12:22.360818 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:22.360924 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 01:12:22.362272 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:22.361855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39129 (local address 127.0.0.1:58836)
0929 01:12:22.362023 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:22.362124 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 01:12:22.409313 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 25988
I20260929 01:12:22.423256 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:22.422614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39129 (local address 127.0.0.1:58838)
0929 01:12:22.422895 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:22.423016 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 01:12:22.424401 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:22.423984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39129 (local address 127.0.0.1:58842)
0929 01:12:22.424175 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:22.424262 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 01:12:22.428287 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 26118
I20260929 01:12:22.435931 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:22.428994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48718 (local address 127.5.198.195:36101)
0929 01:12:22.429584 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:22.429591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:22.429613 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:12:22.429686 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:22.429694 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:22.429812 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:22.429925 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:22.430602 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:22.431425 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:22.432783 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:22.433222 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:22.433374 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:22.434447 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:22.434456 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:22.434470 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:22.435215 (+ 745us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:22.435261 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:22.435280 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:22.435408 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:22.435519 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 01:12:22.435615 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441}
I20260929 01:12:22.436801 26240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:22.505241 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:22.504722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39129 (local address 127.0.0.1:58846)
0929 01:12:22.504932 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:22.505062 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 01:12:22.506299 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:22.505906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39129 (local address 127.0.0.1:58858)
0929 01:12:22.506121 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:22.506187 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 01:12:22.558728 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 26118
I20260929 01:12:22.579947 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 25767
I20260929 01:12:22.581079 25828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:22.690367 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:22.689559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.193:39129 (local address 127.0.0.1:58860)
0929 01:12:22.689969 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:22.690078 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.193:39129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229}
I20260929 01:12:22.690555 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 25767
2026-09-29T01:12:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2786 ms)
[----------] 6 tests from EncryptionPolicyTest (21555 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:22 dist-test-slave-41cl krb5kdc[26297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:22 dist-test-slave-41cl krb5kdc[26297](info): set up 2 sockets
Sep 29 01:12:22 dist-test-slave-41cl krb5kdc[26297](info): commencing operation
krb5kdc: starting...
W20260929 01:12:23.788930 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:23 dist-test-slave-41cl krb5kdc[26297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:12:24.054708 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40309
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:24.394277 26310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:24.394636 26310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:24.394806 26310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:24.402303 26310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:24.402478 26310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:24.402588 26310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:24.402702 26310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:24.402786 26310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:24.412948 26310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40309
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:24.414278 26310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:24.415655 26310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:24.428138 26318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:24.428197 26316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:24.428700 26315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:24.429103 26310 server_base.cc:1034] running on GCE node
I20260929 01:12:24.429970 26310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:24.431440 26310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:24.432639 26310 hybrid_clock.cc:648] HybridClock initialized: now 1790644344432590 us; error 40 us; skew 500 ppm
I20260929 01:12:24.433068 26310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:24 dist-test-slave-41cl krb5kdc[26297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:24.436414 26310 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:12:24.438051 26310 webserver.cc:460] Webserver started at http://127.0.0.1:40681/ using document root <none> and password file <none>
I20260929 01:12:24.438731 26310 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:24.438833 26310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:24.439193 26310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:24.442173 26310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "74727eb273924e83b18a5a5e45317402"
format_stamp: "Formatted at 2026-09-29 01:12:24 on dist-test-slave-41cl"
server_key: "56f1382027fafeddb7b32d01eb26cb27"
server_key_iv: "c2392fec6c4e619c8bc12e3c3f561f0e"
server_key_version: "encryptionkey@0"
I20260929 01:12:24.442934 26310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "74727eb273924e83b18a5a5e45317402"
format_stamp: "Formatted at 2026-09-29 01:12:24 on dist-test-slave-41cl"
server_key: "56f1382027fafeddb7b32d01eb26cb27"
server_key_iv: "c2392fec6c4e619c8bc12e3c3f561f0e"
server_key_version: "encryptionkey@0"
I20260929 01:12:24.448683 26310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:12:24.452283 26325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:24.453729 26310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:12:24.453920 26310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "74727eb273924e83b18a5a5e45317402"
format_stamp: "Formatted at 2026-09-29 01:12:24 on dist-test-slave-41cl"
server_key: "56f1382027fafeddb7b32d01eb26cb27"
server_key_iv: "c2392fec6c4e619c8bc12e3c3f561f0e"
server_key_version: "encryptionkey@0"
I20260929 01:12:24.454116 26310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:24.466773 26310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:24.471052 26310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:24.471338 26310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:24.490227 26310 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40309
I20260929 01:12:24.490361 26376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40309 every 8 connection(s)
I20260929 01:12:24.491688 26310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:24.495698 26377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:24.499188 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 26310
I20260929 01:12:24.499404 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:24.500238 5915 external_mini_cluster.cc:1428] Setting key 7cdb120a0dd0d4f79d99072bc10ce10d
I20260929 01:12:24.508949 26377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402: Bootstrap starting.
I20260929 01:12:24.512832 26377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:24.514197 26377 log.cc:824] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:12:24 dist-test-slave-41cl krb5kdc[26297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 01:12:24.517598 26377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402: No bootstrap required, opened a new log
I20260929 01:12:24.525293 26380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:24.502239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59759 (local address 127.0.0.1:40309)
0929 01:12:24.503004 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:24.503015 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:24.503058 (+ 43us) server_negotiation.cc:407] Connection header received
0929 01:12:24.504739 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:24.504761 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:24.505167 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:24.505496 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:24.507341 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:24.508748 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:24.510491 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:24.511198 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:24.511326 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:24.516542 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:24.516554 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:24.516574 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:24.516629 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:24.520702 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:24.521500 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:24.521511 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:24.521525 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:24.521739 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:24.522166 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:24.522175 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:24.522179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:24.522846 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:24.523178 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:24.523548 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 01:12:24.523885 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":376,"threads_started":1}
I20260929 01:12:24.528216 26377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74727eb273924e83b18a5a5e45317402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40309 } }
I20260929 01:12:24.528658 26377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:24.528805 26377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74727eb273924e83b18a5a5e45317402, State: Initialized, Role: FOLLOWER
I20260929 01:12:24.529459 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:24.529426 26377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74727eb273924e83b18a5a5e45317402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40309 } }
I20260929 01:12:24.529721 26377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:24.529861 26377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:24.530036 26377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:24.532140 26377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74727eb273924e83b18a5a5e45317402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40309 } }
I20260929 01:12:24.532699 26377 leader_election.cc:302] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74727eb273924e83b18a5a5e45317402; no voters:
I20260929 01:12:24.533335 26377 leader_election.cc:288] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:24.533468 26386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:24.535880 26377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:24.535936 26386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [term 1 LEADER]: Becoming Leader. State: Replica: 74727eb273924e83b18a5a5e45317402, State: Running, Role: LEADER
I20260929 01:12:24.536536 26386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74727eb273924e83b18a5a5e45317402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40309 } }
I20260929 01:12:24.539633 26388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74727eb273924e83b18a5a5e45317402. Latest consensus state: current_term: 1 leader_uuid: "74727eb273924e83b18a5a5e45317402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74727eb273924e83b18a5a5e45317402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40309 } } }
I20260929 01:12:24.540071 26388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:24.540657 26387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74727eb273924e83b18a5a5e45317402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74727eb273924e83b18a5a5e45317402" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40309 } } }
I20260929 01:12:24.540983 26387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:24.549050 26395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:24.553961 26380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:24.534939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33453 (local address 127.0.0.1:40309)
0929 01:12:24.535100 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:24.535105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:24.535147 (+ 42us) server_negotiation.cc:407] Connection header received
0929 01:12:24.535200 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:24.535205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:24.535272 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:24.535370 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:24.537456 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:24.538779 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:24.540623 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:24.541561 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:24.541712 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:24.546023 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:24.546033 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:24.546041 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:24.546092 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:24.550322 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:24.551464 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:24.551469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:24.551472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:24.551624 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:24.552252 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:24.552257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:24.552260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:24.552725 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:24.553002 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:24.553419 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 01:12:24.553593 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:12:24.554888 26395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:24.564440 26395 catalog_manager.cc:1347] Generated new cluster ID: 16f34b4e9a5b477483caed021228da37
I20260929 01:12:24.564594 26395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:24.577033 26395 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:24.578420 26395 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:24.585695 26395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74727eb273924e83b18a5a5e45317402: Generated new TSK 0
I20260929 01:12:24.586463 26395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:24.592242 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 26310
I20260929 01:12:24.594704 26372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:24.685671 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 26310
W20260929 01:12:24.703315 26295 connection.cc:535] client connection to 127.0.0.1:40309 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40309): Connection reset by peer (error 104)
2026-09-29T01:12:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1995 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:24 dist-test-slave-41cl krb5kdc[26410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:24 dist-test-slave-41cl krb5kdc[26410](info): set up 2 sockets
Sep 29 01:12:24 dist-test-slave-41cl krb5kdc[26410](info): commencing operation
krb5kdc: starting...
W20260929 01:12:25.800904 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:25 dist-test-slave-41cl krb5kdc[26410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:12:26.060568 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:26.393996 26423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:26.394345 26423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:26.394536 26423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:26.401865 26423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:26.402029 26423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:26.402145 26423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:26.402266 26423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:26.402362 26423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:26.412168 26423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40505
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:26.413515 26423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:26.414839 26423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:26.427170 26428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:26.427120 26429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:26.427120 26431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:26.428187 26423 server_base.cc:1034] running on GCE node
I20260929 01:12:26.429090 26423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:26.430531 26423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:26.431762 26423 hybrid_clock.cc:648] HybridClock initialized: now 1790644346431702 us; error 55 us; skew 500 ppm
I20260929 01:12:26.432152 26423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:26 dist-test-slave-41cl krb5kdc[26410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:26.435607 26423 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:12:26.437306 26423 webserver.cc:460] Webserver started at http://127.0.0.1:42071/ using document root <none> and password file <none>
I20260929 01:12:26.438022 26423 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:26.438139 26423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:26.438405 26423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:26.441679 26423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "ed71c6a2bb114b1c9a5a82c3bac51d26"
format_stamp: "Formatted at 2026-09-29 01:12:26 on dist-test-slave-41cl"
server_key: "b9f0fcad47176264a4c330df1f359505"
server_key_iv: "d65728c10e4f6b5beff1a1c22bea0c4a"
server_key_version: "encryptionkey@0"
I20260929 01:12:26.442456 26423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "ed71c6a2bb114b1c9a5a82c3bac51d26"
format_stamp: "Formatted at 2026-09-29 01:12:26 on dist-test-slave-41cl"
server_key: "b9f0fcad47176264a4c330df1f359505"
server_key_iv: "d65728c10e4f6b5beff1a1c22bea0c4a"
server_key_version: "encryptionkey@0"
I20260929 01:12:26.447999 26423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:12:26.451783 26438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:26.453361 26423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:12:26.453563 26423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "ed71c6a2bb114b1c9a5a82c3bac51d26"
format_stamp: "Formatted at 2026-09-29 01:12:26 on dist-test-slave-41cl"
server_key: "b9f0fcad47176264a4c330df1f359505"
server_key_iv: "d65728c10e4f6b5beff1a1c22bea0c4a"
server_key_version: "encryptionkey@0"
I20260929 01:12:26.453784 26423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:26.477173 26423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:26.481597 26423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:26.481881 26423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:26.500638 26423 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40505
I20260929 01:12:26.500689 26489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40505 every 8 connection(s)
I20260929 01:12:26.502022 26423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:26.504343 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 26423
I20260929 01:12:26.504560 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:26.505048 5915 external_mini_cluster.cc:1428] Setting key 93dad6876d3d484e8ee91af5351fbf2f
I20260929 01:12:26.505895 26490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:26.519584 26490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26: Bootstrap starting.
I20260929 01:12:26.519784 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:26.506734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46945 (local address 127.0.0.1:40505)
0929 01:12:26.507677 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:26.507692 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:26.508032 (+ 340us) server_negotiation.cc:407] Connection header received
0929 01:12:26.510640 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:26.510661 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:26.511056 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:26.511373 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:26.512004 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:26.513436 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:26.515234 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:26.515864 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:26.516038 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:26.517409 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:26.517417 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:26.517430 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:26.518014 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:26.518108 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:26.518134 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:26.518265 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:26.518600 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 01:12:26.518780 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":386,"threads_started":1}
I20260929 01:12:26.522898 26490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:26.523576 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:26.524056 26490 log.cc:824] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:26.526706 26490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26: No bootstrap required, opened a new log
I20260929 01:12:26.535331 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:26.528371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35121 (local address 127.0.0.1:40505)
0929 01:12:26.528621 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:26.528630 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:26.528921 (+ 291us) server_negotiation.cc:407] Connection header received
0929 01:12:26.529098 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:26.529102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:26.529166 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:26.529254 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:26.529812 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:26.530668 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:26.532600 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:26.533147 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:26.533268 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:26.534464 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:26.534469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:26.534473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:26.534495 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:26.534531 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:26.534547 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:26.534643 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:26.534858 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 01:12:26.534942 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 01:12:26.536298 26490 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed71c6a2bb114b1c9a5a82c3bac51d26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } }
I20260929 01:12:26.536744 26490 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:26.536859 26490 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed71c6a2bb114b1c9a5a82c3bac51d26, State: Initialized, Role: FOLLOWER
I20260929 01:12:26.537413 26490 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed71c6a2bb114b1c9a5a82c3bac51d26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } }
I20260929 01:12:26.537653 26490 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:26.537792 26490 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:26.537971 26490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:26.540052 26490 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed71c6a2bb114b1c9a5a82c3bac51d26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } }
I20260929 01:12:26.540622 26490 leader_election.cc:302] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed71c6a2bb114b1c9a5a82c3bac51d26; no voters:
I20260929 01:12:26.541258 26490 leader_election.cc:288] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:26.541371 26500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:26.543337 26490 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:26.544919 26500 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [term 1 LEADER]: Becoming Leader. State: Replica: ed71c6a2bb114b1c9a5a82c3bac51d26, State: Running, Role: LEADER
I20260929 01:12:26.545599 26500 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed71c6a2bb114b1c9a5a82c3bac51d26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } }
I20260929 01:12:26.548421 26502 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed71c6a2bb114b1c9a5a82c3bac51d26. Latest consensus state: current_term: 1 leader_uuid: "ed71c6a2bb114b1c9a5a82c3bac51d26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed71c6a2bb114b1c9a5a82c3bac51d26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } } }
I20260929 01:12:26.548849 26502 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:26.548741 26501 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed71c6a2bb114b1c9a5a82c3bac51d26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed71c6a2bb114b1c9a5a82c3bac51d26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } } }
I20260929 01:12:26.549117 26501 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26 [sys.catalog]: This master's current role is: LEADER
W20260929 01:12:26.555433 26514 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:12:26.555567 26514 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:12:26.555727 26515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:26.557940 26515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:26.566672 26515 catalog_manager.cc:1347] Generated new cluster ID: 838e879f68284d639113fb4ee8e5a06c
I20260929 01:12:26.566780 26515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:26.584702 26515 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:26.585989 26515 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:26.594465 26515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed71c6a2bb114b1c9a5a82c3bac51d26: Generated new TSK 0
I20260929 01:12:26.595230 26515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:26.600976 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 26423
W20260929 01:12:26.602000 26485 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46945
W20260929 01:12:26.602731 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 26423: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:12:26.602905 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 26423
W20260929 01:12:26.619899 26408 connection.cc:535] client connection to 127.0.0.1:40505 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40505): Connection reset by peer (error 104)
2026-09-29T01:12:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:26 dist-test-slave-41cl krb5kdc[26521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:26 dist-test-slave-41cl krb5kdc[26521](info): set up 2 sockets
Sep 29 01:12:26 dist-test-slave-41cl krb5kdc[26521](info): commencing operation
krb5kdc: starting...
W20260929 01:12:27.708853 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:27 dist-test-slave-41cl krb5kdc[26521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:12:27.964774 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:28.296213 26534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:28.296607 26534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:28.296784 26534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:28.305111 26534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:28.305260 26534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:28.305375 26534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:28.305460 26534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:28.305541 26534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:28.315912 26534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33429
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:28.317255 26534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:28.318580 26534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:28.330200 26539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:28.330163 26540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:28.330222 26542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:28.330818 26534 server_base.cc:1034] running on GCE node
I20260929 01:12:28.331544 26534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:28.332895 26534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:28.334154 26534 hybrid_clock.cc:648] HybridClock initialized: now 1790644348334099 us; error 46 us; skew 500 ppm
I20260929 01:12:28.334513 26534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:28 dist-test-slave-41cl krb5kdc[26521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:28.343189 26534 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:12:28.345103 26534 webserver.cc:460] Webserver started at http://127.0.0.1:38189/ using document root <none> and password file <none>
I20260929 01:12:28.345817 26534 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:28.345945 26534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:28.346258 26534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:28.349324 26534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "b8bcbf6bf49343e8bb4c268a76733ceb"
format_stamp: "Formatted at 2026-09-29 01:12:28 on dist-test-slave-41cl"
server_key: "e7605f190438a303b913eb6396e661e2"
server_key_iv: "370738fb45d53814863e28945e85bac5"
server_key_version: "encryptionkey@0"
I20260929 01:12:28.350191 26534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "b8bcbf6bf49343e8bb4c268a76733ceb"
format_stamp: "Formatted at 2026-09-29 01:12:28 on dist-test-slave-41cl"
server_key: "e7605f190438a303b913eb6396e661e2"
server_key_iv: "370738fb45d53814863e28945e85bac5"
server_key_version: "encryptionkey@0"
I20260929 01:12:28.355510 26534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 01:12:28.359172 26549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:28.360612 26534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:12:28.360793 26534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "b8bcbf6bf49343e8bb4c268a76733ceb"
format_stamp: "Formatted at 2026-09-29 01:12:28 on dist-test-slave-41cl"
server_key: "e7605f190438a303b913eb6396e661e2"
server_key_iv: "370738fb45d53814863e28945e85bac5"
server_key_version: "encryptionkey@0"
I20260929 01:12:28.360993 26534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:28.367108 26534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:28.371163 26534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:28.371553 26534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:28.390041 26534 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33429
I20260929 01:12:28.390125 26600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33429 every 8 connection(s)
I20260929 01:12:28.391426 26534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:28.395071 26601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:28.399197 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 26534
I20260929 01:12:28.399369 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:28.399820 5915 external_mini_cluster.cc:1428] Setting key cd4a75332e1289299339c149bccc4bc8
I20260929 01:12:28.405913 26601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb: Bootstrap starting.
I20260929 01:12:28.407678 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:28.401456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49889 (local address 127.0.0.1:33429)
0929 01:12:28.402201 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:28.402211 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:28.402249 (+ 38us) server_negotiation.cc:407] Connection header received
0929 01:12:28.404301 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:28.404322 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:28.404756 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:28.405143 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:28.406189 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:28.406196 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:28.406207 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:28.406220 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:28.406280 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:28.406299 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:28.406472 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:28.406589 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 01:12:28.406793 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":341,"threads_started":1}
I20260929 01:12:28.409121 26601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:28.410245 26601 log.cc:824] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:28.411113 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:28.412968 26601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb: No bootstrap required, opened a new log
I20260929 01:12:28.417680 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:28.415379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59495 (local address 127.0.0.1:33429)
0929 01:12:28.415569 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:28.415574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:28.415856 (+ 282us) server_negotiation.cc:407] Connection header received
0929 01:12:28.416014 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:28.416018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:28.416058 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:28.416165 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:28.417038 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:28.417042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:28.417045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:28.417057 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:28.417091 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:28.417104 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:28.417215 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:28.417351 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 01:12:28.417395 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:12:28.421456 26601 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bcbf6bf49343e8bb4c268a76733ceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33429 } }
I20260929 01:12:28.421759 26601 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:28.421828 26601 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8bcbf6bf49343e8bb4c268a76733ceb, State: Initialized, Role: FOLLOWER
I20260929 01:12:28.422297 26601 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bcbf6bf49343e8bb4c268a76733ceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33429 } }
I20260929 01:12:28.422497 26601 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:28.422586 26601 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:28.422710 26601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:28.425096 26601 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bcbf6bf49343e8bb4c268a76733ceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33429 } }
I20260929 01:12:28.425510 26601 leader_election.cc:302] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8bcbf6bf49343e8bb4c268a76733ceb; no voters:
I20260929 01:12:28.426079 26601 leader_election.cc:288] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:28.426182 26611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:28.426788 26611 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [term 1 LEADER]: Becoming Leader. State: Replica: b8bcbf6bf49343e8bb4c268a76733ceb, State: Running, Role: LEADER
I20260929 01:12:28.427301 26611 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bcbf6bf49343e8bb4c268a76733ceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33429 } }
I20260929 01:12:28.428202 26601 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:28.431386 26612 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8bcbf6bf49343e8bb4c268a76733ceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bcbf6bf49343e8bb4c268a76733ceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33429 } } }
I20260929 01:12:28.431864 26612 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:28.431732 26613 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8bcbf6bf49343e8bb4c268a76733ceb. Latest consensus state: current_term: 1 leader_uuid: "b8bcbf6bf49343e8bb4c268a76733ceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bcbf6bf49343e8bb4c268a76733ceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33429 } } }
I20260929 01:12:28.432124 26613 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:28.432941 26621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:28.438602 26621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:28.447649 26621 catalog_manager.cc:1347] Generated new cluster ID: 9d6f97515cd04c008738e0c93b5611f9
I20260929 01:12:28.447758 26621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:28.469372 26621 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:28.469780 26621 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:28.481948 26621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8bcbf6bf49343e8bb4c268a76733ceb: Generated new TSK 0
I20260929 01:12:28.482795 26621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:28.489215 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 26534
W20260929 01:12:28.490193 26596 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49889
W20260929 01:12:28.490991 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 26534: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:12:28.491148 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 26534
2026-09-29T01:12:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1887 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:28 dist-test-slave-41cl krb5kdc[26632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:28 dist-test-slave-41cl krb5kdc[26632](info): set up 2 sockets
Sep 29 01:12:28 dist-test-slave-41cl krb5kdc[26632](info): commencing operation
krb5kdc: starting...
W20260929 01:12:29.592895 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:29 dist-test-slave-41cl krb5kdc[26632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:12:29.833819 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35175
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:30.160266 26645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:30.160612 26645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:30.160772 26645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:30.167937 26645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:30.168092 26645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:30.168208 26645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:30.168306 26645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:30.168390 26645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:30.178045 26645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:30.179389 26645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:30.180768 26645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:30.192445 26650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:30.192499 26653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:30.192816 26651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:30.193938 26645 server_base.cc:1034] running on GCE node
I20260929 01:12:30.194736 26645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:30.196118 26645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:30.197325 26645 hybrid_clock.cc:648] HybridClock initialized: now 1790644350197287 us; error 55 us; skew 500 ppm
I20260929 01:12:30.197666 26645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:30 dist-test-slave-41cl krb5kdc[26632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:30.200646 26645 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:12:30.202085 26645 webserver.cc:460] Webserver started at http://127.0.0.1:33499/ using document root <none> and password file <none>
I20260929 01:12:30.202737 26645 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:30.202857 26645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:30.203153 26645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:30.206037 26645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "e0d7e93faede4a48aca8823d919fb016"
format_stamp: "Formatted at 2026-09-29 01:12:30 on dist-test-slave-41cl"
server_key: "e4e368b8638d3f2213eaef6b8f28e5ab"
server_key_iv: "a5875e520de648af3219a4706e797933"
server_key_version: "encryptionkey@0"
I20260929 01:12:30.206794 26645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "e0d7e93faede4a48aca8823d919fb016"
format_stamp: "Formatted at 2026-09-29 01:12:30 on dist-test-slave-41cl"
server_key: "e4e368b8638d3f2213eaef6b8f28e5ab"
server_key_iv: "a5875e520de648af3219a4706e797933"
server_key_version: "encryptionkey@0"
I20260929 01:12:30.211776 26645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 01:12:30.215147 26660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:30.216398 26645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:12:30.216631 26645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "e0d7e93faede4a48aca8823d919fb016"
format_stamp: "Formatted at 2026-09-29 01:12:30 on dist-test-slave-41cl"
server_key: "e4e368b8638d3f2213eaef6b8f28e5ab"
server_key_iv: "a5875e520de648af3219a4706e797933"
server_key_version: "encryptionkey@0"
I20260929 01:12:30.216843 26645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:30.234969 26645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:30.239430 26645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:30.239681 26645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:30.257339 26645 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35175
I20260929 01:12:30.257362 26711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35175 every 8 connection(s)
I20260929 01:12:30.258721 26645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:30.262404 26712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:30.267901 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 26645
I20260929 01:12:30.268131 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:30.268620 5915 external_mini_cluster.cc:1428] Setting key cec9429249a7150839c0c541a502cf81
I20260929 01:12:30.273077 26712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016: Bootstrap starting.
I20260929 01:12:30.276811 26712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:30.278123 26712 log.cc:824] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:30.280930 26712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016: No bootstrap required, opened a new log
Sep 29 01:12:30 dist-test-slave-41cl krb5kdc[26632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 01:12:30.290783 26712 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d7e93faede4a48aca8823d919fb016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } }
I20260929 01:12:30.291235 26712 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:30.291344 26712 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d7e93faede4a48aca8823d919fb016, State: Initialized, Role: FOLLOWER
I20260929 01:12:30.292058 26712 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d7e93faede4a48aca8823d919fb016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } }
I20260929 01:12:30.292374 26712 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:30.292567 26712 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:30.292757 26712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:30.294399 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:30.270589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43737 (local address 127.0.0.1:35175)
0929 01:12:30.271539 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:30.271553 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:30.271933 (+ 380us) server_negotiation.cc:407] Connection header received
0929 01:12:30.274411 (+ 2478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:30.274439 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:30.274962 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:30.275403 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:30.276785 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:30.278814 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:30.280070 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:30.280847 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:30.281042 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:30.285382 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:30.285394 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:30.285413 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:30.285460 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:30.289695 (+ 4235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:30.290471 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:30.290479 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:30.290487 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:30.290686 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:30.291084 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:30.291089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:30.291091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:30.291816 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:30.292122 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:30.292551 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 01:12:30.293031 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":428,"threads_started":1}
I20260929 01:12:30.296079 26712 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d7e93faede4a48aca8823d919fb016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } }
I20260929 01:12:30.296720 26712 leader_election.cc:302] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0d7e93faede4a48aca8823d919fb016; no voters:
I20260929 01:12:30.297436 26712 leader_election.cc:288] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:30.297639 26717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:30.298875 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:30.299526 26712 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:30.302881 26717 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [term 1 LEADER]: Becoming Leader. State: Replica: e0d7e93faede4a48aca8823d919fb016, State: Running, Role: LEADER
I20260929 01:12:30.303704 26717 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d7e93faede4a48aca8823d919fb016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } }
I20260929 01:12:30.307480 26718 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0d7e93faede4a48aca8823d919fb016" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d7e93faede4a48aca8823d919fb016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } } }
I20260929 01:12:30.307965 26718 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:30.308633 26719 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0d7e93faede4a48aca8823d919fb016. Latest consensus state: current_term: 1 leader_uuid: "e0d7e93faede4a48aca8823d919fb016" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d7e93faede4a48aca8823d919fb016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35175 } } }
I20260929 01:12:30.308967 26719 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:30.318816 26731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:30.323904 26731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:30.334393 26731 catalog_manager.cc:1347] Generated new cluster ID: 103cee00c33c47999f6787abd3513471
I20260929 01:12:30.334573 26731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:30.335816 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:30.315774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39487 (local address 127.0.0.1:35175)
0929 01:12:30.316002 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:30.316013 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:30.319973 (+ 3960us) server_negotiation.cc:407] Connection header received
0929 01:12:30.320207 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:30.320216 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:30.320307 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:30.320430 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:30.322365 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:30.323765 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:30.325514 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:30.326244 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:30.326345 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:30.329421 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:30.329426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:30.329430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:30.329462 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:30.332667 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:30.333317 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:30.333321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:30.333324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:30.333439 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:30.333867 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:30.333872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:30.333875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:30.334271 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:30.334518 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:30.335137 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 01:12:30.335380 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 01:12:30.351629 26731 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:30.352978 26731 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:30.363620 26731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0d7e93faede4a48aca8823d919fb016: Generated new TSK 0
I20260929 01:12:30.364864 26731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:30.379217 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 26645
I20260929 01:12:30.380640 26707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:30.472589 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 26645
2026-09-29T01:12:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1981 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:30 dist-test-slave-41cl krb5kdc[26745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:30 dist-test-slave-41cl krb5kdc[26745](info): set up 2 sockets
Sep 29 01:12:30 dist-test-slave-41cl krb5kdc[26745](info): commencing operation
krb5kdc: starting...
W20260929 01:12:31.577018 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:31 dist-test-slave-41cl krb5kdc[26745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:12:31.828929 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:32.163175 26758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:32.163523 26758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:32.163690 26758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:32.170855 26758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:32.171015 26758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:32.171119 26758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:32.171226 26758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:32.171299 26758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:32.181187 26758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:32.182492 26758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:32.183878 26758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:32.195573 26763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:32.195678 26764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:32.195677 26766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:32.196339 26758 server_base.cc:1034] running on GCE node
I20260929 01:12:32.197166 26758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:32.198470 26758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:32.199679 26758 hybrid_clock.cc:648] HybridClock initialized: now 1790644352199642 us; error 35 us; skew 500 ppm
I20260929 01:12:32.200030 26758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:32 dist-test-slave-41cl krb5kdc[26745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:32.203215 26758 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:12:32.204731 26758 webserver.cc:460] Webserver started at http://127.0.0.1:45281/ using document root <none> and password file <none>
I20260929 01:12:32.205368 26758 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:32.205466 26758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:32.205713 26758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:32.208616 26758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "a7084cab7bad406ea009b1181085ccea"
format_stamp: "Formatted at 2026-09-29 01:12:32 on dist-test-slave-41cl"
server_key: "8068a8083c3b7eda3b7a8d971b2ca814"
server_key_iv: "46cd74c621635fd704dbd3be89fda45d"
server_key_version: "encryptionkey@0"
I20260929 01:12:32.209322 26758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "a7084cab7bad406ea009b1181085ccea"
format_stamp: "Formatted at 2026-09-29 01:12:32 on dist-test-slave-41cl"
server_key: "8068a8083c3b7eda3b7a8d971b2ca814"
server_key_iv: "46cd74c621635fd704dbd3be89fda45d"
server_key_version: "encryptionkey@0"
I20260929 01:12:32.214258 26758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:12:32.217622 26773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:32.218916 26758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:12:32.219092 26758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "a7084cab7bad406ea009b1181085ccea"
format_stamp: "Formatted at 2026-09-29 01:12:32 on dist-test-slave-41cl"
server_key: "8068a8083c3b7eda3b7a8d971b2ca814"
server_key_iv: "46cd74c621635fd704dbd3be89fda45d"
server_key_version: "encryptionkey@0"
I20260929 01:12:32.219295 26758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:32.232558 26758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:32.236860 26758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:32.237206 26758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:32.256340 26758 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41939
I20260929 01:12:32.256374 26824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41939 every 8 connection(s)
I20260929 01:12:32.257807 26758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:32.261493 26825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:32.262567 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 26758
I20260929 01:12:32.262753 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:32.263362 5915 external_mini_cluster.cc:1428] Setting key aa428222161154f01150a7bd3106823e
I20260929 01:12:32.274288 26825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea: Bootstrap starting.
I20260929 01:12:32.276504 26828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:32.265197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52557 (local address 127.0.0.1:41939)
0929 01:12:32.265986 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:32.265997 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:32.266043 (+ 46us) server_negotiation.cc:407] Connection header received
0929 01:12:32.267664 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:32.267687 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:32.268141 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:32.268676 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:32.269681 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:32.270892 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:32.272225 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:32.272918 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:32.273117 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:32.274309 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:32.274316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:32.274328 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:32.274880 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:32.274960 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:32.274983 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:32.275111 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:32.275322 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 01:12:32.275565 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":369,"threads_started":1}
I20260929 01:12:32.277458 26825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:32.278473 26825 log.cc:824] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:32.280072 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:32.281734 26825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea: No bootstrap required, opened a new log
I20260929 01:12:32.292008 26828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:32.285366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56167 (local address 127.0.0.1:41939)
0929 01:12:32.285607 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:32.285616 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:32.286008 (+ 392us) server_negotiation.cc:407] Connection header received
0929 01:12:32.286176 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:32.286180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:32.286244 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:32.286348 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:32.286964 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:32.287705 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:32.289574 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:32.290066 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:32.290185 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:32.291316 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:32.291321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:32.291324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:32.291340 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:32.291377 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:32.291392 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:32.291465 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:32.291614 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 01:12:32.291679 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 01:12:32.292176 26825 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7084cab7bad406ea009b1181085ccea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } }
I20260929 01:12:32.292634 26825 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:32.292766 26825 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7084cab7bad406ea009b1181085ccea, State: Initialized, Role: FOLLOWER
I20260929 01:12:32.293385 26825 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7084cab7bad406ea009b1181085ccea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } }
I20260929 01:12:32.293627 26825 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:32.293774 26825 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:32.293944 26825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:32.296104 26825 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7084cab7bad406ea009b1181085ccea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } }
I20260929 01:12:32.296645 26825 leader_election.cc:302] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7084cab7bad406ea009b1181085ccea; no voters:
I20260929 01:12:32.297211 26825 leader_election.cc:288] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:32.297353 26835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:32.298022 26835 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [term 1 LEADER]: Becoming Leader. State: Replica: a7084cab7bad406ea009b1181085ccea, State: Running, Role: LEADER
I20260929 01:12:32.298645 26835 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7084cab7bad406ea009b1181085ccea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } }
I20260929 01:12:32.299252 26825 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:32.301565 26836 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7084cab7bad406ea009b1181085ccea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7084cab7bad406ea009b1181085ccea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } } }
I20260929 01:12:32.301932 26836 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:32.301649 26837 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7084cab7bad406ea009b1181085ccea. Latest consensus state: current_term: 1 leader_uuid: "a7084cab7bad406ea009b1181085ccea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7084cab7bad406ea009b1181085ccea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } } }
I20260929 01:12:32.302075 26837 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:32.305300 26841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:32.311681 26841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:32.322108 26841 catalog_manager.cc:1347] Generated new cluster ID: b46880cef8124dffbb0f9152eea51480
I20260929 01:12:32.322319 26841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:32.337165 26841 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:32.338544 26841 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:32.355690 26841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7084cab7bad406ea009b1181085ccea: Generated new TSK 0
I20260929 01:12:32.356933 26841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:32.373383 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 26758
W20260929 01:12:32.374565 26820 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52557
W20260929 01:12:32.375279 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 26758: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:12:32.375439 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 26758
2026-09-29T01:12:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1902 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:32 dist-test-slave-41cl krb5kdc[26856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:32 dist-test-slave-41cl krb5kdc[26856](info): set up 2 sockets
Sep 29 01:12:32 dist-test-slave-41cl krb5kdc[26856](info): commencing operation
krb5kdc: starting...
W20260929 01:12:33.476940 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:33 dist-test-slave-41cl krb5kdc[26856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:12:33.728439 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:34.054790 26869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:34.055068 26869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:34.055233 26869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:34.062444 26869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:34.062574 26869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:34.062685 26869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:34.062785 26869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:34.062870 26869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:34.072634 26869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:34.073902 26869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:34.075165 26869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:34.086959 26874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:34.086896 26877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:34.086989 26875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:34.088081 26869 server_base.cc:1034] running on GCE node
I20260929 01:12:34.088855 26869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:34.090233 26869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:34.091460 26869 hybrid_clock.cc:648] HybridClock initialized: now 1790644354091427 us; error 53 us; skew 500 ppm
I20260929 01:12:34.091821 26869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:34 dist-test-slave-41cl krb5kdc[26856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:34.101635 26869 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:12:34.103287 26869 webserver.cc:460] Webserver started at http://127.0.0.1:43745/ using document root <none> and password file <none>
I20260929 01:12:34.104034 26869 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:34.104195 26869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:34.104609 26869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:34.107831 26869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "9d516f6b07da4ad089c09956be3cffef"
format_stamp: "Formatted at 2026-09-29 01:12:34 on dist-test-slave-41cl"
server_key: "56dd10adb777d8b6c876b975f64c57ca"
server_key_iv: "598d6451eef6edc9d200b70ca8189ee6"
server_key_version: "encryptionkey@0"
I20260929 01:12:34.108745 26869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "9d516f6b07da4ad089c09956be3cffef"
format_stamp: "Formatted at 2026-09-29 01:12:34 on dist-test-slave-41cl"
server_key: "56dd10adb777d8b6c876b975f64c57ca"
server_key_iv: "598d6451eef6edc9d200b70ca8189ee6"
server_key_version: "encryptionkey@0"
I20260929 01:12:34.114347 26869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:12:34.117908 26884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:34.119299 26869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:12:34.119508 26869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "9d516f6b07da4ad089c09956be3cffef"
format_stamp: "Formatted at 2026-09-29 01:12:34 on dist-test-slave-41cl"
server_key: "56dd10adb777d8b6c876b975f64c57ca"
server_key_iv: "598d6451eef6edc9d200b70ca8189ee6"
server_key_version: "encryptionkey@0"
I20260929 01:12:34.119714 26869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:34.125753 26869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:34.130013 26869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:34.130299 26869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:34.148882 26869 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34553
I20260929 01:12:34.148969 26935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34553 every 8 connection(s)
I20260929 01:12:34.150246 26869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:34.151712 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 26869
I20260929 01:12:34.151901 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:34.152477 5915 external_mini_cluster.cc:1428] Setting key 7cf73a879d5df29ce25c935fdc667de0
I20260929 01:12:34.154842 26936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:34.161005 26939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:34.154217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39565 (local address 127.0.0.1:34553)
0929 01:12:34.154975 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:34.154989 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:34.155038 (+ 49us) server_negotiation.cc:407] Connection header received
0929 01:12:34.157301 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:34.157325 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:34.157766 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:34.158201 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:34.159146 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:34.159154 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:34.159171 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:34.159197 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:34.159277 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:34.159314 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:34.159529 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:34.159658 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 01:12:34.159846 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":301,"threads_started":1}
I20260929 01:12:34.165082 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:34.168288 26936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef: Bootstrap starting.
I20260929 01:12:34.172204 26936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:34.172909 26939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:34.170324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35167 (local address 127.0.0.1:34553)
0929 01:12:34.170497 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:34.170509 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:34.170966 (+ 457us) server_negotiation.cc:407] Connection header received
0929 01:12:34.171101 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:34.171104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:34.171148 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:34.171230 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:34.172170 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:34.172173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:34.172176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:34.172193 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:34.172228 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:34.172242 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:34.172395 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:34.172571 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 01:12:34.172620 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 01:12:34.173481 26936 log.cc:824] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:34.176046 26936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef: No bootstrap required, opened a new log
I20260929 01:12:34.182965 26936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d516f6b07da4ad089c09956be3cffef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34553 } }
I20260929 01:12:34.183305 26936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:34.183398 26936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d516f6b07da4ad089c09956be3cffef, State: Initialized, Role: FOLLOWER
I20260929 01:12:34.183974 26936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d516f6b07da4ad089c09956be3cffef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34553 } }
I20260929 01:12:34.184195 26936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:34.184345 26936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:34.184549 26936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:34.186615 26936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d516f6b07da4ad089c09956be3cffef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34553 } }
I20260929 01:12:34.187083 26936 leader_election.cc:302] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d516f6b07da4ad089c09956be3cffef; no voters:
I20260929 01:12:34.187634 26936 leader_election.cc:288] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:34.187759 26946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:34.188410 26946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [term 1 LEADER]: Becoming Leader. State: Replica: 9d516f6b07da4ad089c09956be3cffef, State: Running, Role: LEADER
I20260929 01:12:34.188947 26946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d516f6b07da4ad089c09956be3cffef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34553 } }
I20260929 01:12:34.189683 26936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:34.191956 26947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d516f6b07da4ad089c09956be3cffef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d516f6b07da4ad089c09956be3cffef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34553 } } }
I20260929 01:12:34.191968 26948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d516f6b07da4ad089c09956be3cffef. Latest consensus state: current_term: 1 leader_uuid: "9d516f6b07da4ad089c09956be3cffef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d516f6b07da4ad089c09956be3cffef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34553 } } }
I20260929 01:12:34.192384 26948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:34.192384 26947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:34.194224 26953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:34.198462 26953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:34.209316 26963 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:12:34.209496 26953 catalog_manager.cc:1347] Generated new cluster ID: 29303b0859db4a9fa9119cce3d5f2488
I20260929 01:12:34.210166 26953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:34.225533 26953 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:34.225999 26953 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:34.239660 26953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d516f6b07da4ad089c09956be3cffef: Generated new TSK 0
I20260929 01:12:34.240746 26953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:34.247997 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 26869
W20260929 01:12:34.249228 26931 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39565
W20260929 01:12:34.249974 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 26869: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:12:34.250128 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 26869
2026-09-29T01:12:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1874 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:34 dist-test-slave-41cl krb5kdc[26967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:34 dist-test-slave-41cl krb5kdc[26967](info): set up 2 sockets
Sep 29 01:12:34 dist-test-slave-41cl krb5kdc[26967](info): commencing operation
krb5kdc: starting...
W20260929 01:12:35.412917 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:35 dist-test-slave-41cl krb5kdc[26967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:12:35.668286 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:35.997099 26980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:35.997372 26980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:35.997466 26980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:36.004539 26980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:36.004642 26980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:36.004686 26980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:36.004745 26980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:36.004789 26980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:36.014654 26980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:36.015870 26980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:36.017190 26980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:36.028457 26985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:36.029145 26988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:36.029171 26986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:36.029757 26980 server_base.cc:1034] running on GCE node
I20260929 01:12:36.030514 26980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:36.031824 26980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:36.033053 26980 hybrid_clock.cc:648] HybridClock initialized: now 1790644356033010 us; error 60 us; skew 500 ppm
I20260929 01:12:36.033411 26980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:36 dist-test-slave-41cl krb5kdc[26967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:36.036631 26980 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:12:36.038128 26980 webserver.cc:460] Webserver started at http://127.0.0.1:36235/ using document root <none> and password file <none>
I20260929 01:12:36.038769 26980 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:36.038865 26980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:36.039111 26980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:36.042012 26980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "7eeb1d7332c1442a8019c187cd0ff1d1"
format_stamp: "Formatted at 2026-09-29 01:12:36 on dist-test-slave-41cl"
server_key: "8abd62d89c6bf9ca73a30bd8332378eb"
server_key_iv: "8ddb0d8779222524370133fae65badc6"
server_key_version: "encryptionkey@0"
I20260929 01:12:36.042716 26980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "7eeb1d7332c1442a8019c187cd0ff1d1"
format_stamp: "Formatted at 2026-09-29 01:12:36 on dist-test-slave-41cl"
server_key: "8abd62d89c6bf9ca73a30bd8332378eb"
server_key_iv: "8ddb0d8779222524370133fae65badc6"
server_key_version: "encryptionkey@0"
I20260929 01:12:36.047669 26980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:12:36.050937 26995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:36.052225 26980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:12:36.052407 26980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "7eeb1d7332c1442a8019c187cd0ff1d1"
format_stamp: "Formatted at 2026-09-29 01:12:36 on dist-test-slave-41cl"
server_key: "8abd62d89c6bf9ca73a30bd8332378eb"
server_key_iv: "8ddb0d8779222524370133fae65badc6"
server_key_version: "encryptionkey@0"
I20260929 01:12:36.052644 26980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:36.102139 26980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:36.106565 26980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:36.106865 26980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:36.125375 26980 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46093
I20260929 01:12:36.125404 27046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46093 every 8 connection(s)
I20260929 01:12:36.126673 26980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:36.130210 27047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:36.131979 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 26980
I20260929 01:12:36.132172 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:36.132730 5915 external_mini_cluster.cc:1428] Setting key a09748f2b641d3e0598921f2190952c1
I20260929 01:12:36.144148 27047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1: Bootstrap starting.
Sep 29 01:12:36 dist-test-slave-41cl krb5kdc[26967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 01:12:36.147704 27047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:36.148833 27047 log.cc:824] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:36.151467 27047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1: No bootstrap required, opened a new log
I20260929 01:12:36.154796 27050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:36.134466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:37549 (local address 127.0.0.1:46093)
0929 01:12:36.135314 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:36.135328 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:36.135369 (+ 41us) server_negotiation.cc:407] Connection header received
0929 01:12:36.136979 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:36.136999 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:36.137375 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:36.137680 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:36.139357 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:36.140705 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:36.142426 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:36.143024 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:36.143171 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:36.147705 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:36.147712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:36.147725 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:36.147762 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:36.150858 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:36.151605 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:36.151620 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:36.151626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:36.151775 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:36.152241 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:36.152245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:36.152247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:36.152776 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:36.153107 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:36.153581 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 01:12:36.153871 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":364,"threads_started":1}
I20260929 01:12:36.158490 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:36.160153 27047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb1d7332c1442a8019c187cd0ff1d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46093 } }
I20260929 01:12:36.160588 27047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:36.160712 27047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eeb1d7332c1442a8019c187cd0ff1d1, State: Initialized, Role: FOLLOWER
I20260929 01:12:36.161362 27047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb1d7332c1442a8019c187cd0ff1d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46093 } }
I20260929 01:12:36.161674 27047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:36.161839 27047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:36.162034 27047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:36.164899 27047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb1d7332c1442a8019c187cd0ff1d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46093 } }
I20260929 01:12:36.165373 27047 leader_election.cc:302] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eeb1d7332c1442a8019c187cd0ff1d1; no voters:
I20260929 01:12:36.165987 27047 leader_election.cc:288] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:36.166093 27057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:36.167287 27057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [term 1 LEADER]: Becoming Leader. State: Replica: 7eeb1d7332c1442a8019c187cd0ff1d1, State: Running, Role: LEADER
I20260929 01:12:36.167906 27057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb1d7332c1442a8019c187cd0ff1d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46093 } }
I20260929 01:12:36.168273 27047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:36.174031 27058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eeb1d7332c1442a8019c187cd0ff1d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb1d7332c1442a8019c187cd0ff1d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46093 } } }
I20260929 01:12:36.174522 27058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:36.175217 27059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eeb1d7332c1442a8019c187cd0ff1d1. Latest consensus state: current_term: 1 leader_uuid: "7eeb1d7332c1442a8019c187cd0ff1d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeb1d7332c1442a8019c187cd0ff1d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46093 } } }
I20260929 01:12:36.175628 27059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:36.181123 27050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:36.163758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:48207 (local address 127.0.0.1:46093)
0929 01:12:36.163937 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:36.163943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:36.164328 (+ 385us) server_negotiation.cc:407] Connection header received
0929 01:12:36.164442 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:36.164447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:36.164541 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:36.164649 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:36.166024 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:36.167274 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:36.168798 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:36.169249 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:36.169383 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:36.171792 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:36.171797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:36.171801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:36.171828 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:36.175900 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:36.177291 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:36.177299 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:36.177305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:36.177473 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:36.178196 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:36.178204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:36.178209 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:36.178699 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:36.180125 (+ 1426us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:36.180351 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 01:12:36.180608 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:12:36.185412 27063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:36.191002 27063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:36.199663 27063 catalog_manager.cc:1347] Generated new cluster ID: f60877e764774ec9b80fdf1bacd92114
I20260929 01:12:36.199838 27063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:36.232386 27063 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:36.233786 27063 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:36.240442 27063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eeb1d7332c1442a8019c187cd0ff1d1: Generated new TSK 0
I20260929 01:12:36.241255 27063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:36.250284 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 26980
I20260929 01:12:36.251993 27042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:36.345309 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 26980
W20260929 01:12:36.360051 26965 connection.cc:535] client connection to 127.0.0.1:46093 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46093): Connection reset by peer (error 104)
2026-09-29T01:12:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2092 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:36 dist-test-slave-41cl krb5kdc[27080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:36 dist-test-slave-41cl krb5kdc[27080](info): set up 2 sockets
Sep 29 01:12:36 dist-test-slave-41cl krb5kdc[27080](info): commencing operation
krb5kdc: starting...
W20260929 01:12:37.441030 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:37 dist-test-slave-41cl krb5kdc[27080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:12:37.695581 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:38.021641 27093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:38.021993 27093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:38.022181 27093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:38.030275 27093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:38.030431 27093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:38.030543 27093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:38.030643 27093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:38.030719 27093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:38.040920 27093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:38.042263 27093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:38.043659 27093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:38.055912 27099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:38.055943 27101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:38.056185 27098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:38.056684 27093 server_base.cc:1034] running on GCE node
I20260929 01:12:38.057476 27093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:38.058821 27093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:38.060048 27093 hybrid_clock.cc:648] HybridClock initialized: now 1790644358059994 us; error 52 us; skew 500 ppm
I20260929 01:12:38.060380 27093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:38 dist-test-slave-41cl krb5kdc[27080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:38.063596 27093 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:12:38.065057 27093 webserver.cc:460] Webserver started at http://127.0.0.1:34753/ using document root <none> and password file <none>
I20260929 01:12:38.065723 27093 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:38.065869 27093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:38.066181 27093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:38.069149 27093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "f2a0458842e847b981eb623810a2568e"
format_stamp: "Formatted at 2026-09-29 01:12:38 on dist-test-slave-41cl"
server_key: "962f18b87e295ea9e942b3e20c48c50b"
server_key_iv: "0cde4ddaa73981c0a6975401cbebd95a"
server_key_version: "encryptionkey@0"
I20260929 01:12:38.069897 27093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "f2a0458842e847b981eb623810a2568e"
format_stamp: "Formatted at 2026-09-29 01:12:38 on dist-test-slave-41cl"
server_key: "962f18b87e295ea9e942b3e20c48c50b"
server_key_iv: "0cde4ddaa73981c0a6975401cbebd95a"
server_key_version: "encryptionkey@0"
I20260929 01:12:38.074905 27093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:12:38.078336 27108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:38.079573 27093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:12:38.079756 27093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "f2a0458842e847b981eb623810a2568e"
format_stamp: "Formatted at 2026-09-29 01:12:38 on dist-test-slave-41cl"
server_key: "962f18b87e295ea9e942b3e20c48c50b"
server_key_iv: "0cde4ddaa73981c0a6975401cbebd95a"
server_key_version: "encryptionkey@0"
I20260929 01:12:38.079943 27093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:38.094641 27093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:38.098985 27093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:38.099267 27093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:38.118059 27093 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35351
I20260929 01:12:38.118104 27159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35351 every 8 connection(s)
I20260929 01:12:38.119415 27093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:38.123276 27160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:38.128893 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 27093
I20260929 01:12:38.129088 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:38.129652 5915 external_mini_cluster.cc:1428] Setting key bc05329254037483c36899c82662ef21
I20260929 01:12:38.135129 27160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e: Bootstrap starting.
I20260929 01:12:38.138950 27160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:38.140182 27160 log.cc:824] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:38.143474 27160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e: No bootstrap required, opened a new log
I20260929 01:12:38.143975 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:38.131454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:44199 (local address 127.0.0.1:35351)
0929 01:12:38.132249 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:38.132260 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:38.132304 (+ 44us) server_negotiation.cc:407] Connection header received
0929 01:12:38.134061 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:38.134083 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:38.134470 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:38.134777 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:38.135408 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:38.136859 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:38.139186 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:38.139847 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:38.139978 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:38.141587 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:38.141598 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:38.141618 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:38.142175 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:38.142247 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:38.142272 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:38.142393 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:38.142721 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 01:12:38.142923 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":367,"threads_started":1}
I20260929 01:12:38.148204 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:38.154810 27160 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a0458842e847b981eb623810a2568e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35351 } }
I20260929 01:12:38.155261 27160 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:38.155395 27160 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a0458842e847b981eb623810a2568e, State: Initialized, Role: FOLLOWER
I20260929 01:12:38.156162 27160 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a0458842e847b981eb623810a2568e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35351 } }
I20260929 01:12:38.156457 27160 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:38.156651 27160 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:38.156836 27160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:38.159852 27160 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a0458842e847b981eb623810a2568e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35351 } }
I20260929 01:12:38.160504 27160 leader_election.cc:302] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2a0458842e847b981eb623810a2568e; no voters:
I20260929 01:12:38.161257 27160 leader_election.cc:288] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:38.161369 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:38.153889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:53495 (local address 127.0.0.1:35351)
0929 01:12:38.154086 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:38.154093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:38.154552 (+ 459us) server_negotiation.cc:407] Connection header received
0929 01:12:38.154701 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:38.154707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:38.154795 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:38.154909 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:38.155501 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:38.156807 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:38.158155 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:38.158952 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:38.159081 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:38.160318 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:38.160325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:38.160330 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:38.160353 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:38.160417 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:38.160439 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:38.160648 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:38.160836 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 01:12:38.160891 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:12:38.161366 27170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:38.163110 27170 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [term 1 LEADER]: Becoming Leader. State: Replica: f2a0458842e847b981eb623810a2568e, State: Running, Role: LEADER
I20260929 01:12:38.163712 27170 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a0458842e847b981eb623810a2568e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35351 } }
I20260929 01:12:38.165555 27170 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2a0458842e847b981eb623810a2568e. Latest consensus state: current_term: 1 leader_uuid: "f2a0458842e847b981eb623810a2568e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a0458842e847b981eb623810a2568e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35351 } } }
I20260929 01:12:38.165882 27170 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:38.167200 27172 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2a0458842e847b981eb623810a2568e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a0458842e847b981eb623810a2568e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35351 } } }
I20260929 01:12:38.167629 27172 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:38.168030 27160 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:38.168236 27174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:38.172435 27174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:38.182288 27174 catalog_manager.cc:1347] Generated new cluster ID: 656cbed643ac420ea9dcce83f0c9ac67
I20260929 01:12:38.182427 27174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:38.194365 27174 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:38.196108 27174 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:38.204531 27174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2a0458842e847b981eb623810a2568e: Generated new TSK 0
I20260929 01:12:38.205379 27174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:38.220340 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 27093
W20260929 01:12:38.221493 27155 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.43:44199
W20260929 01:12:38.222306 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 27093: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:12:38.222486 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 27093
W20260929 01:12:38.236830 27078 connection.cc:535] client connection to 127.0.0.1:35351 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35351): Connection reset by peer (error 104)
2026-09-29T01:12:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1876 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:38 dist-test-slave-41cl krb5kdc[27191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:38 dist-test-slave-41cl krb5kdc[27191](info): set up 2 sockets
Sep 29 01:12:38 dist-test-slave-41cl krb5kdc[27191](info): commencing operation
krb5kdc: starting...
W20260929 01:12:39.324962 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:39 dist-test-slave-41cl krb5kdc[27191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:12:39.575954 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:39.903281 27204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:39.903551 27204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:39.903735 27204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:39.910827 27204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:39.910931 27204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:39.911047 27204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:39.911156 27204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:39.911223 27204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:39.921006 27204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:39.922272 27204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:39.923563 27204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:39.936219 27209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:39.936244 27212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:39.936156 27210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:39.936991 27204 server_base.cc:1034] running on GCE node
I20260929 01:12:39.937793 27204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:39.939213 27204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:39.940529 27204 hybrid_clock.cc:648] HybridClock initialized: now 1790644359940443 us; error 67 us; skew 500 ppm
I20260929 01:12:39.940912 27204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:39 dist-test-slave-41cl krb5kdc[27191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:39.950188 27204 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:12:39.951752 27204 webserver.cc:460] Webserver started at http://127.0.0.1:35893/ using document root <none> and password file <none>
I20260929 01:12:39.952548 27204 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:39.952718 27204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:39.953040 27204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:39.955950 27204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "707bd9e907e1460c8085e546cc3a53fc"
format_stamp: "Formatted at 2026-09-29 01:12:39 on dist-test-slave-41cl"
server_key: "fb14647c090d74864d8e81db570c6a48"
server_key_iv: "e531e310aae56b500d075141031b6674"
server_key_version: "encryptionkey@0"
I20260929 01:12:39.956792 27204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "707bd9e907e1460c8085e546cc3a53fc"
format_stamp: "Formatted at 2026-09-29 01:12:39 on dist-test-slave-41cl"
server_key: "fb14647c090d74864d8e81db570c6a48"
server_key_iv: "e531e310aae56b500d075141031b6674"
server_key_version: "encryptionkey@0"
I20260929 01:12:39.961951 27204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 01:12:39.965445 27219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:39.966840 27204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:12:39.967041 27204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "707bd9e907e1460c8085e546cc3a53fc"
format_stamp: "Formatted at 2026-09-29 01:12:39 on dist-test-slave-41cl"
server_key: "fb14647c090d74864d8e81db570c6a48"
server_key_iv: "e531e310aae56b500d075141031b6674"
server_key_version: "encryptionkey@0"
I20260929 01:12:39.967247 27204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:39.973107 27204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:39.977495 27204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:39.977820 27204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:39.998034 27204 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33137
I20260929 01:12:39.998075 27270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33137 every 8 connection(s)
I20260929 01:12:39.999643 27204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:40.003590 27271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:40.009032 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 27204
I20260929 01:12:40.009222 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:40.009680 5915 external_mini_cluster.cc:1428] Setting key d13e4e5623275eac67a4abf17d264062
I20260929 01:12:40.015604 27271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc: Bootstrap starting.
I20260929 01:12:40.018790 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:40.011602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:35339 (local address 127.0.0.1:33137)
0929 01:12:40.012441 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:40.012478 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:40.012544 (+ 66us) server_negotiation.cc:407] Connection header received
0929 01:12:40.014925 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:40.014951 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:40.015393 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:40.015835 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:40.016838 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:40.016848 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:40.016863 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:40.016880 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:40.016948 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:40.016974 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:40.017191 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:40.017340 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 01:12:40.017572 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":348,"threads_started":1}
I20260929 01:12:40.019553 27271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:40.020725 27271 log.cc:824] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:40.022505 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:40.023952 27271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc: No bootstrap required, opened a new log
I20260929 01:12:40.030181 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:40.027708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:38489 (local address 127.0.0.1:33137)
0929 01:12:40.027893 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:40.027899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:40.028346 (+ 447us) server_negotiation.cc:407] Connection header received
0929 01:12:40.028489 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:40.028494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:40.028558 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:40.028664 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:40.029540 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:40.029551 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:40.029556 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:40.029586 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:40.029637 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:40.029655 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:40.029766 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:40.029837 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 01:12:40.029896 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:12:40.033210 27271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707bd9e907e1460c8085e546cc3a53fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33137 } }
I20260929 01:12:40.033550 27271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:40.033649 27271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 707bd9e907e1460c8085e546cc3a53fc, State: Initialized, Role: FOLLOWER
I20260929 01:12:40.034317 27271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707bd9e907e1460c8085e546cc3a53fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33137 } }
I20260929 01:12:40.034551 27271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:40.034678 27271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:40.034847 27271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:40.037276 27271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707bd9e907e1460c8085e546cc3a53fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33137 } }
I20260929 01:12:40.037751 27271 leader_election.cc:302] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 707bd9e907e1460c8085e546cc3a53fc; no voters:
I20260929 01:12:40.038327 27271 leader_election.cc:288] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:40.038491 27281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:40.039222 27281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [term 1 LEADER]: Becoming Leader. State: Replica: 707bd9e907e1460c8085e546cc3a53fc, State: Running, Role: LEADER
I20260929 01:12:40.039811 27281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707bd9e907e1460c8085e546cc3a53fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33137 } }
I20260929 01:12:40.040338 27271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:40.043043 27283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 707bd9e907e1460c8085e546cc3a53fc. Latest consensus state: current_term: 1 leader_uuid: "707bd9e907e1460c8085e546cc3a53fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707bd9e907e1460c8085e546cc3a53fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33137 } } }
I20260929 01:12:40.043444 27283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:40.044193 27282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "707bd9e907e1460c8085e546cc3a53fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707bd9e907e1460c8085e546cc3a53fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33137 } } }
I20260929 01:12:40.044623 27282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:40.045128 27288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:40.050108 27288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:40.059844 27288 catalog_manager.cc:1347] Generated new cluster ID: fdb9f4c46aae408098756651e86ac311
I20260929 01:12:40.059955 27288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:40.092978 27288 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:40.093403 27288 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:40.102206 27288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 707bd9e907e1460c8085e546cc3a53fc: Generated new TSK 0
I20260929 01:12:40.103113 27288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:40.112609 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 27204
W20260929 01:12:40.113613 27266 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.43:35339
W20260929 01:12:40.114236 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 27204: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:12:40.114383 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 27204
2026-09-29T01:12:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1894 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:40 dist-test-slave-41cl krb5kdc[27302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:40 dist-test-slave-41cl krb5kdc[27302](info): set up 2 sockets
Sep 29 01:12:40 dist-test-slave-41cl krb5kdc[27302](info): commencing operation
krb5kdc: starting...
W20260929 01:12:41.233279 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:41 dist-test-slave-41cl krb5kdc[27302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:12:41.479830 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:41.807337 27315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:41.807621 27315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:41.807713 27315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:41.814802 27315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:41.814899 27315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:41.814944 27315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:41.815003 27315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:41.815049 27315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:41.824720 27315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:41.826028 27315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:41.827354 27315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:41.838951 27320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:41.838999 27321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:41.839423 27323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:41.839851 27315 server_base.cc:1034] running on GCE node
I20260929 01:12:41.840626 27315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:41.842013 27315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:41.843247 27315 hybrid_clock.cc:648] HybridClock initialized: now 1790644361843206 us; error 33 us; skew 500 ppm
I20260929 01:12:41.843588 27315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:41 dist-test-slave-41cl krb5kdc[27302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:41.846621 27315 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:12:41.848074 27315 webserver.cc:460] Webserver started at http://127.0.0.1:36611/ using document root <none> and password file <none>
I20260929 01:12:41.848802 27315 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:41.848946 27315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:41.849206 27315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:41.852108 27315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "09fba860135e4d4d9d3b22bedf3b257a"
format_stamp: "Formatted at 2026-09-29 01:12:41 on dist-test-slave-41cl"
server_key: "e9178f635e31bb45fdda890e2b9b529e"
server_key_iv: "0fc6bd25da32c404f8f55b35a3eebb9a"
server_key_version: "encryptionkey@0"
I20260929 01:12:41.852854 27315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "09fba860135e4d4d9d3b22bedf3b257a"
format_stamp: "Formatted at 2026-09-29 01:12:41 on dist-test-slave-41cl"
server_key: "e9178f635e31bb45fdda890e2b9b529e"
server_key_iv: "0fc6bd25da32c404f8f55b35a3eebb9a"
server_key_version: "encryptionkey@0"
I20260929 01:12:41.859113 27315 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:12:41.863873 27330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:41.865527 27315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:12:41.865685 27315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "09fba860135e4d4d9d3b22bedf3b257a"
format_stamp: "Formatted at 2026-09-29 01:12:41 on dist-test-slave-41cl"
server_key: "e9178f635e31bb45fdda890e2b9b529e"
server_key_iv: "0fc6bd25da32c404f8f55b35a3eebb9a"
server_key_version: "encryptionkey@0"
I20260929 01:12:41.865865 27315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:41.882449 27315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:41.886708 27315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:41.886979 27315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:41.905536 27315 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46525
I20260929 01:12:41.905565 27381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46525 every 8 connection(s)
I20260929 01:12:41.907135 27315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:41.911032 27382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:41.913203 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 27315
I20260929 01:12:41.913434 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:41.913905 5915 external_mini_cluster.cc:1428] Setting key c33da549741b916fd7f0a32401b178b4
I20260929 01:12:41.924839 27382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a: Bootstrap starting.
I20260929 01:12:41.928416 27382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:41.929435 27382 log.cc:824] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:12:41 dist-test-slave-41cl krb5kdc[27302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 01:12:41.932926 27382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a: No bootstrap required, opened a new log
I20260929 01:12:41.939949 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:41.915721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:33951 (local address 127.0.0.1:46525)
0929 01:12:41.916667 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:41.916682 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:41.916737 (+ 55us) server_negotiation.cc:407] Connection header received
0929 01:12:41.919166 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:41.919203 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:41.919734 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:41.920208 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:41.921627 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:41.923677 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:41.925277 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:41.926270 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:41.926466 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:41.931076 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:41.931088 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:41.931119 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:41.931166 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:41.935186 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:41.936027 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:41.936036 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:41.936044 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:41.936261 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:41.936780 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:41.936787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:41.936795 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:41.937561 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:41.937890 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:41.938237 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 01:12:41.938592 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":410,"threads_started":1}
I20260929 01:12:41.943816 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:41.943635 27382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09fba860135e4d4d9d3b22bedf3b257a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46525 } }
I20260929 01:12:41.944021 27382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:41.944135 27382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09fba860135e4d4d9d3b22bedf3b257a, State: Initialized, Role: FOLLOWER
I20260929 01:12:41.944763 27382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09fba860135e4d4d9d3b22bedf3b257a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46525 } }
I20260929 01:12:41.945066 27382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:41.945222 27382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:41.945393 27382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:41.947714 27382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09fba860135e4d4d9d3b22bedf3b257a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46525 } }
I20260929 01:12:41.948195 27382 leader_election.cc:302] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09fba860135e4d4d9d3b22bedf3b257a; no voters:
I20260929 01:12:41.948871 27382 leader_election.cc:288] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:41.948975 27391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:41.949661 27391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [term 1 LEADER]: Becoming Leader. State: Replica: 09fba860135e4d4d9d3b22bedf3b257a, State: Running, Role: LEADER
I20260929 01:12:41.950238 27391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09fba860135e4d4d9d3b22bedf3b257a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46525 } }
I20260929 01:12:41.951925 27391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09fba860135e4d4d9d3b22bedf3b257a. Latest consensus state: current_term: 1 leader_uuid: "09fba860135e4d4d9d3b22bedf3b257a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09fba860135e4d4d9d3b22bedf3b257a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46525 } } }
I20260929 01:12:41.952248 27391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:41.952802 27394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09fba860135e4d4d9d3b22bedf3b257a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09fba860135e4d4d9d3b22bedf3b257a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46525 } } }
I20260929 01:12:41.953212 27394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:41.957233 27382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:41.959230 27396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:41.965402 27396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:41.969223 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:41.948447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:52231 (local address 127.0.0.1:46525)
0929 01:12:41.948666 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:41.948675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:41.949281 (+ 606us) server_negotiation.cc:407] Connection header received
0929 01:12:41.949543 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:41.949552 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:41.949666 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:41.949767 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:41.952419 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:41.954030 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:41.956153 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:41.956874 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:41.956977 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:41.960681 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:41.960688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:41.960693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:41.960734 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:41.964672 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:41.965781 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:41.965786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:41.965789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:41.965931 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:41.966528 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:41.966533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:41.966536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:41.966946 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:41.968157 (+ 1211us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:41.968560 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 01:12:41.968744 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 01:12:41.975589 27396 catalog_manager.cc:1347] Generated new cluster ID: dbaa32caa97648f6a47a9ad5e909c734
I20260929 01:12:41.975706 27396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:41.997176 27396 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:41.998430 27396 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:42.011904 27396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09fba860135e4d4d9d3b22bedf3b257a: Generated new TSK 0
I20260929 01:12:42.012817 27396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:42.018324 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 27315
I20260929 01:12:42.019929 27377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:42.110391 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 27315
W20260929 01:12:42.127514 27300 connection.cc:535] client connection to 127.0.0.1:46525 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46525): Connection reset by peer (error 104)
2026-09-29T01:12:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1996 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:42 dist-test-slave-41cl krb5kdc[27414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:42 dist-test-slave-41cl krb5kdc[27414](info): set up 2 sockets
Sep 29 01:12:42 dist-test-slave-41cl krb5kdc[27414](info): commencing operation
krb5kdc: starting...
W20260929 01:12:43.221014 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:43 dist-test-slave-41cl krb5kdc[27414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:12:43.475445 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:43.798091 27427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:43.798363 27427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:43.798527 27427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:43.805660 27427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:43.805795 27427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:43.805913 27427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:43.806020 27427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:43.806129 27427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:43.815935 27427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:43.817269 27427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:43.818529 27427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:43.830355 27433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:43.830312 27435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:43.830403 27432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:43.831420 27427 server_base.cc:1034] running on GCE node
I20260929 01:12:43.832186 27427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:43.833652 27427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:43.834890 27427 hybrid_clock.cc:648] HybridClock initialized: now 1790644363834831 us; error 55 us; skew 500 ppm
I20260929 01:12:43.835238 27427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:43 dist-test-slave-41cl krb5kdc[27414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:43.838557 27427 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:12:43.839957 27427 webserver.cc:460] Webserver started at http://127.0.0.1:33831/ using document root <none> and password file <none>
I20260929 01:12:43.840691 27427 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:43.840823 27427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:43.841141 27427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:43.844002 27427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "c61543af7d0144c1b8ac8a1323b9d9b0"
format_stamp: "Formatted at 2026-09-29 01:12:43 on dist-test-slave-41cl"
server_key: "ec7b7866a4f13de96e1ed27e3633622c"
server_key_iv: "85d01bf33f70cc82bf45319ea4e8b5cc"
server_key_version: "encryptionkey@0"
I20260929 01:12:43.844808 27427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "c61543af7d0144c1b8ac8a1323b9d9b0"
format_stamp: "Formatted at 2026-09-29 01:12:43 on dist-test-slave-41cl"
server_key: "ec7b7866a4f13de96e1ed27e3633622c"
server_key_iv: "85d01bf33f70cc82bf45319ea4e8b5cc"
server_key_version: "encryptionkey@0"
I20260929 01:12:43.849818 27427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 01:12:43.853329 27442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:43.854803 27427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:12:43.854947 27427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "c61543af7d0144c1b8ac8a1323b9d9b0"
format_stamp: "Formatted at 2026-09-29 01:12:43 on dist-test-slave-41cl"
server_key: "ec7b7866a4f13de96e1ed27e3633622c"
server_key_iv: "85d01bf33f70cc82bf45319ea4e8b5cc"
server_key_version: "encryptionkey@0"
I20260929 01:12:43.855270 27427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:43.883397 27427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:43.887603 27427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:43.887905 27427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:43.907716 27427 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36155
I20260929 01:12:43.907752 27493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36155 every 8 connection(s)
I20260929 01:12:43.909159 27427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:43.909583 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 27427
I20260929 01:12:43.909762 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:43.910384 5915 external_mini_cluster.cc:1428] Setting key c651524c8edb17c34434f8541c194806
I20260929 01:12:43.913980 27494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:43.923342 27497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:43.912195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:42761 (local address 127.0.0.1:36155)
0929 01:12:43.912990 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:43.913003 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:43.913045 (+ 42us) server_negotiation.cc:407] Connection header received
0929 01:12:43.914918 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:43.914938 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:43.915306 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:43.915685 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:43.916238 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:43.917647 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:43.918963 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:43.919628 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:43.919816 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:43.921172 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:43.921179 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:43.921193 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:43.921714 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:43.921784 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:43.921807 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:43.921980 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:43.922246 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 01:12:43.922408 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":376,"threads_started":1}
I20260929 01:12:43.925638 27494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0: Bootstrap starting.
I20260929 01:12:43.927371 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:43.929122 27494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:43.930163 27494 log.cc:824] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:43.933774 27494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0: No bootstrap required, opened a new log
I20260929 01:12:43.938521 27497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:43.931955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:42377 (local address 127.0.0.1:36155)
0929 01:12:43.932132 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:43.932139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:43.932400 (+ 261us) server_negotiation.cc:407] Connection header received
0929 01:12:43.932595 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:43.932602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:43.932686 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:43.932810 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:43.933263 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:43.934434 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:43.935733 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:43.936525 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:43.936687 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:43.937700 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:43.937708 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:43.937713 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:43.937741 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:43.937811 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:43.937836 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:43.937971 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:43.938107 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 01:12:43.938170 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:12:43.942957 27494 raft_consensus.cc:348] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61543af7d0144c1b8ac8a1323b9d9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } }
I20260929 01:12:43.943315 27494 raft_consensus.cc:374] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:43.943405 27494 raft_consensus.cc:729] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61543af7d0144c1b8ac8a1323b9d9b0, State: Initialized, Role: FOLLOWER
I20260929 01:12:43.943915 27494 consensus_queue.cc:260] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61543af7d0144c1b8ac8a1323b9d9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } }
I20260929 01:12:43.944118 27494 raft_consensus.cc:388] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:43.944223 27494 raft_consensus.cc:482] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:43.944358 27494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:43.946964 27494 raft_consensus.cc:504] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61543af7d0144c1b8ac8a1323b9d9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } }
I20260929 01:12:43.947453 27494 leader_election.cc:302] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c61543af7d0144c1b8ac8a1323b9d9b0; no voters:
I20260929 01:12:43.947999 27494 leader_election.cc:288] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:43.948181 27504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:43.948945 27504 raft_consensus.cc:686] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [term 1 LEADER]: Becoming Leader. State: Replica: c61543af7d0144c1b8ac8a1323b9d9b0, State: Running, Role: LEADER
I20260929 01:12:43.949467 27504 consensus_queue.cc:237] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61543af7d0144c1b8ac8a1323b9d9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } }
I20260929 01:12:43.950198 27494 sys_catalog.cc:564] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:43.952104 27505 sys_catalog.cc:455] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c61543af7d0144c1b8ac8a1323b9d9b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61543af7d0144c1b8ac8a1323b9d9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } } }
I20260929 01:12:43.952425 27505 sys_catalog.cc:458] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:43.952710 27506 sys_catalog.cc:455] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c61543af7d0144c1b8ac8a1323b9d9b0. Latest consensus state: current_term: 1 leader_uuid: "c61543af7d0144c1b8ac8a1323b9d9b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61543af7d0144c1b8ac8a1323b9d9b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } } }
I20260929 01:12:43.953104 27506 sys_catalog.cc:458] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:43.954619 27509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:43.959439 27509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:43.967561 27509 catalog_manager.cc:1347] Generated new cluster ID: bab10b55ea61407aaea0c9c3c6e46e3a
I20260929 01:12:43.967670 27509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:43.971800 27521 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:12:43.990447 27509 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:43.991679 27509 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:44.010617 27509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c61543af7d0144c1b8ac8a1323b9d9b0: Generated new TSK 0
I20260929 01:12:44.011513 27509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:44.017396 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 27427
W20260929 01:12:44.018617 27489 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.43:42761
W20260929 01:12:44.019335 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 27427: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:12:44.019510 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 27427
W20260929 01:12:44.036782 27412 connection.cc:535] client connection to 127.0.0.1:36155 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36155): Connection reset by peer (error 104)
2026-09-29T01:12:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:44 dist-test-slave-41cl krb5kdc[27525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:44 dist-test-slave-41cl krb5kdc[27525](info): set up 2 sockets
Sep 29 01:12:44 dist-test-slave-41cl krb5kdc[27525](info): commencing operation
krb5kdc: starting...
W20260929 01:12:45.168897 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:45 dist-test-slave-41cl krb5kdc[27525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:12:45.432485 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42255
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:45.760993 27538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:45.761344 27538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:45.761512 27538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:45.768936 27538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:45.769101 27538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:45.769212 27538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:45.769331 27538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:45.769413 27538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:45.779577 27538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42255
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:45.780911 27538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:45.782207 27538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:45.793708 27544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:45.793812 27546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:45.793823 27543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:45.794679 27538 server_base.cc:1034] running on GCE node
I20260929 01:12:45.795397 27538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:45.796811 27538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:45.798046 27538 hybrid_clock.cc:648] HybridClock initialized: now 1790644365798005 us; error 34 us; skew 500 ppm
I20260929 01:12:45.798380 27538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:45 dist-test-slave-41cl krb5kdc[27525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:45.801404 27538 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:12:45.802806 27538 webserver.cc:460] Webserver started at http://127.0.0.1:43723/ using document root <none> and password file <none>
I20260929 01:12:45.803448 27538 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:45.803588 27538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:45.803894 27538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:45.806851 27538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "b56b98df80c742d98246a87fd0335d76"
format_stamp: "Formatted at 2026-09-29 01:12:45 on dist-test-slave-41cl"
server_key: "d9ae6425b16653d42510d27bbdf28afd"
server_key_iv: "b89ecb91b5a5993155034f6e9a9807bc"
server_key_version: "encryptionkey@0"
I20260929 01:12:45.807595 27538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "b56b98df80c742d98246a87fd0335d76"
format_stamp: "Formatted at 2026-09-29 01:12:45 on dist-test-slave-41cl"
server_key: "d9ae6425b16653d42510d27bbdf28afd"
server_key_iv: "b89ecb91b5a5993155034f6e9a9807bc"
server_key_version: "encryptionkey@0"
I20260929 01:12:45.812371 27538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260929 01:12:45.815722 27553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:45.817282 27538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:12:45.817476 27538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "b56b98df80c742d98246a87fd0335d76"
format_stamp: "Formatted at 2026-09-29 01:12:45 on dist-test-slave-41cl"
server_key: "d9ae6425b16653d42510d27bbdf28afd"
server_key_iv: "b89ecb91b5a5993155034f6e9a9807bc"
server_key_version: "encryptionkey@0"
I20260929 01:12:45.817675 27538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:45.823503 27538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:45.827566 27538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:45.827853 27538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:45.846624 27538 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42255
I20260929 01:12:45.846709 27604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42255 every 8 connection(s)
I20260929 01:12:45.848075 27538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:45.851706 27605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:45.855893 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 27538
I20260929 01:12:45.856069 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:45.856676 5915 external_mini_cluster.cc:1428] Setting key f3844e0f9b4c79fe0f3af85197d8a0d7
I20260929 01:12:45.866367 27605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76: Bootstrap starting.
I20260929 01:12:45.868098 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:45.859010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:38371 (local address 127.0.0.1:42255)
0929 01:12:45.861503 (+ 2493us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:45.861526 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:45.861575 (+ 49us) server_negotiation.cc:407] Connection header received
0929 01:12:45.864228 (+ 2653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:45.864262 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:45.864763 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:45.865186 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:45.866509 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:45.866515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:45.866528 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:45.866539 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:45.866589 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:45.866610 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:45.866842 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:45.866981 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 01:12:45.867190 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2203,"thread_start_us":1929,"threads_started":1}
I20260929 01:12:45.869910 27605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:45.871225 27605 log.cc:824] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:45.872424 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:45.874418 27605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76: No bootstrap required, opened a new log
I20260929 01:12:45.879817 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:45.876965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:56625 (local address 127.0.0.1:42255)
0929 01:12:45.877165 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:45.877171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:45.877576 (+ 405us) server_negotiation.cc:407] Connection header received
0929 01:12:45.877720 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:45.877725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:45.877792 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:45.877917 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:45.879039 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:45.879046 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:45.879052 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:45.879075 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:45.879122 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:45.879141 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:45.879273 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:45.879450 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 01:12:45.879521 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:12:45.883332 27605 raft_consensus.cc:348] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56b98df80c742d98246a87fd0335d76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42255 } }
I20260929 01:12:45.883713 27605 raft_consensus.cc:374] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:45.883786 27605 raft_consensus.cc:729] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b56b98df80c742d98246a87fd0335d76, State: Initialized, Role: FOLLOWER
I20260929 01:12:45.884636 27605 consensus_queue.cc:260] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56b98df80c742d98246a87fd0335d76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42255 } }
I20260929 01:12:45.884872 27605 raft_consensus.cc:388] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:45.884982 27605 raft_consensus.cc:482] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:45.885149 27605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:45.887725 27605 raft_consensus.cc:504] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56b98df80c742d98246a87fd0335d76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42255 } }
I20260929 01:12:45.888219 27605 leader_election.cc:302] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b56b98df80c742d98246a87fd0335d76; no voters:
I20260929 01:12:45.888911 27605 leader_election.cc:288] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:45.889200 27615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:45.890641 27615 raft_consensus.cc:686] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [term 1 LEADER]: Becoming Leader. State: Replica: b56b98df80c742d98246a87fd0335d76, State: Running, Role: LEADER
I20260929 01:12:45.891284 27615 consensus_queue.cc:237] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56b98df80c742d98246a87fd0335d76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42255 } }
I20260929 01:12:45.891551 27605 sys_catalog.cc:564] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:45.894677 27616 sys_catalog.cc:455] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b56b98df80c742d98246a87fd0335d76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56b98df80c742d98246a87fd0335d76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42255 } } }
I20260929 01:12:45.894639 27617 sys_catalog.cc:455] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b56b98df80c742d98246a87fd0335d76. Latest consensus state: current_term: 1 leader_uuid: "b56b98df80c742d98246a87fd0335d76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56b98df80c742d98246a87fd0335d76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42255 } } }
I20260929 01:12:45.895358 27617 sys_catalog.cc:458] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:45.895358 27616 sys_catalog.cc:458] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:45.897212 27623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:45.902551 27623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:45.911381 27623 catalog_manager.cc:1347] Generated new cluster ID: 70919cddbe65479e915ad574cd8e0ce3
I20260929 01:12:45.911576 27623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:45.931279 27623 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:45.931696 27623 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:45.940330 27623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b56b98df80c742d98246a87fd0335d76: Generated new TSK 0
I20260929 01:12:45.941457 27623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:45.964186 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 27538
W20260929 01:12:45.965370 27600 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.43:38371
W20260929 01:12:45.966111 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 27538: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:12:45.966260 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 27538
2026-09-29T01:12:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1945 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:46 dist-test-slave-41cl krb5kdc[27636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:46 dist-test-slave-41cl krb5kdc[27636](info): set up 2 sockets
Sep 29 01:12:46 dist-test-slave-41cl krb5kdc[27636](info): commencing operation
krb5kdc: starting...
W20260929 01:12:47.077065 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:47 dist-test-slave-41cl krb5kdc[27636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:12:47.321652 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:44213
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:33687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:44213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:47.648891 27649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:47.649221 27649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:47.649384 27649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:47.656699 27649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:47.656847 27649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:47.656948 27649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:47.657061 27649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:47.657138 27649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:47.666862 27649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:33687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:44213
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:44213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:47.668136 27649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:47.669490 27649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:47.681386 27654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:47.681366 27655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:47.681552 27657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:47.682229 27649 server_base.cc:1034] running on GCE node
I20260929 01:12:47.682957 27649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:47.684332 27649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:47.685570 27649 hybrid_clock.cc:648] HybridClock initialized: now 1790644367685509 us; error 71 us; skew 500 ppm
I20260929 01:12:47.685925 27649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:47 dist-test-slave-41cl krb5kdc[27636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644367, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:47.689052 27649 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:12:47.690542 27649 webserver.cc:460] Webserver started at http://127.5.198.254:39623/ using document root <none> and password file <none>
I20260929 01:12:47.691201 27649 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:47.691345 27649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:47.691645 27649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:47.694633 27649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "ac466a67cf6843ca9a273a7e97771d9e"
format_stamp: "Formatted at 2026-09-29 01:12:47 on dist-test-slave-41cl"
server_key: "f7f9ad7856da4da6fb8d4c7f956db9b2"
server_key_iv: "f23bb7930ef318949265c5f798477c6a"
server_key_version: "encryptionkey@0"
I20260929 01:12:47.695391 27649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "ac466a67cf6843ca9a273a7e97771d9e"
format_stamp: "Formatted at 2026-09-29 01:12:47 on dist-test-slave-41cl"
server_key: "f7f9ad7856da4da6fb8d4c7f956db9b2"
server_key_iv: "f23bb7930ef318949265c5f798477c6a"
server_key_version: "encryptionkey@0"
I20260929 01:12:47.700409 27649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260929 01:12:47.703789 27664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:47.705142 27649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:12:47.705327 27649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "ac466a67cf6843ca9a273a7e97771d9e"
format_stamp: "Formatted at 2026-09-29 01:12:47 on dist-test-slave-41cl"
server_key: "f7f9ad7856da4da6fb8d4c7f956db9b2"
server_key_iv: "f23bb7930ef318949265c5f798477c6a"
server_key_version: "encryptionkey@0"
I20260929 01:12:47.705525 27649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:47.721213 27649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:47.725399 27649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:47.725680 27649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:47.745265 27649 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:44213
I20260929 01:12:47.745285 27715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:44213 every 8 connection(s)
I20260929 01:12:47.746749 27649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:47.750619 27716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:47.755731 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 27649
I20260929 01:12:47.755921 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:47.756376 5915 external_mini_cluster.cc:1428] Setting key ddd387527cf0678cd1a76655bf479398
I20260929 01:12:47.761240 27716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e: Bootstrap starting.
I20260929 01:12:47.765029 27716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:47.766264 27716 log.cc:824] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:47.769752 27716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e: No bootstrap required, opened a new log
Sep 29 01:12:47 dist-test-slave-41cl krb5kdc[27636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:12:47.777704 27716 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac466a67cf6843ca9a273a7e97771d9e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44213 } }
I20260929 01:12:47.778095 27716 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:47.778237 27716 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac466a67cf6843ca9a273a7e97771d9e, State: Initialized, Role: FOLLOWER
I20260929 01:12:47.778990 27716 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac466a67cf6843ca9a273a7e97771d9e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44213 } }
I20260929 01:12:47.779300 27716 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:47.779482 27716 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:47.779668 27716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:47.780514 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:47.758201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40899 (local address 127.5.198.254:44213)
0929 01:12:47.759096 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:47.759106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:47.759149 (+ 43us) server_negotiation.cc:407] Connection header received
0929 01:12:47.760857 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:47.760879 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:47.761266 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:47.761634 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:47.763429 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:47.764785 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:47.766586 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:47.767728 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:47.767856 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:47.772882 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:47.772890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:47.772906 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:47.772941 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:47.776086 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:47.776953 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:47.776960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:47.776969 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:47.777124 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:47.777584 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:47.777588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:47.777590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:47.778138 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:47.778459 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:47.778924 (+ 465us) server_negotiation.cc:299] Negotiation successful
0929 01:12:47.779300 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":415,"threads_started":1}
I20260929 01:12:47.782626 27716 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac466a67cf6843ca9a273a7e97771d9e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44213 } }
I20260929 01:12:47.783182 27716 leader_election.cc:302] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac466a67cf6843ca9a273a7e97771d9e; no voters:
I20260929 01:12:47.784005 27716 leader_election.cc:288] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:47.784063 27721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:47.784596 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:47.785666 27721 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [term 1 LEADER]: Becoming Leader. State: Replica: ac466a67cf6843ca9a273a7e97771d9e, State: Running, Role: LEADER
I20260929 01:12:47.786175 27716 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:47.786139 27721 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac466a67cf6843ca9a273a7e97771d9e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44213 } }
I20260929 01:12:47.789131 27722 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac466a67cf6843ca9a273a7e97771d9e. Latest consensus state: current_term: 1 leader_uuid: "ac466a67cf6843ca9a273a7e97771d9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac466a67cf6843ca9a273a7e97771d9e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44213 } } }
I20260929 01:12:47.789460 27722 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:47.788862 27723 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac466a67cf6843ca9a273a7e97771d9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac466a67cf6843ca9a273a7e97771d9e" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44213 } } }
I20260929 01:12:47.790120 27723 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:47.794337 27728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:47.800693 27728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:47.819059 27728 catalog_manager.cc:1347] Generated new cluster ID: a71ebd8621b24a18a34419c5029c97bb
I20260929 01:12:47.819169 27728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:47.830832 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:47.813390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38017 (local address 127.5.198.254:44213)
0929 01:12:47.813600 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:47.813605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:47.815153 (+ 1548us) server_negotiation.cc:407] Connection header received
0929 01:12:47.815337 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:47.815341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:47.815402 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:47.815530 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:47.817196 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:47.818349 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:47.820118 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:47.820826 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:47.820932 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:47.824028 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:47.824035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:47.824040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:47.824078 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:47.827474 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:47.828328 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:47.828333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:47.828337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:47.828508 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:47.829049 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:47.829054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:47.829057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:47.829424 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:47.829659 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:47.830283 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 01:12:47.830457 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 01:12:47.841308 27728 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:47.843173 27728 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:47.853796 27728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac466a67cf6843ca9a273a7e97771d9e: Generated new TSK 0
I20260929 01:12:47.854809 27728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:47.864308 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 27649
I20260929 01:12:47.865835 27711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:47.955111 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 27649
W20260929 01:12:47.972083 27634 connection.cc:535] client connection to 127.5.198.254:44213 recv error: Network error: failed to read from TLS socket (remote: 127.5.198.254:44213): Connection reset by peer (error 104)
2026-09-29T01:12:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1988 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:48 dist-test-slave-41cl krb5kdc[27749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:48 dist-test-slave-41cl krb5kdc[27749](info): set up 2 sockets
Sep 29 01:12:48 dist-test-slave-41cl krb5kdc[27749](info): commencing operation
krb5kdc: starting...
W20260929 01:12:49.076824 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:49 dist-test-slave-41cl krb5kdc[27749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:12:49.324128 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:33191
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:33191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:49.652626 27762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:49.652920 27762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:49.653106 27762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:49.660594 27762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:49.660748 27762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:49.660864 27762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:49.660989 27762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:49.661085 27762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:49.671208 27762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:33191
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:33191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:49.672590 27762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:49.673903 27762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:49.686179 27768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:49.686131 27770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:49.686432 27767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:49.686969 27762 server_base.cc:1034] running on GCE node
I20260929 01:12:49.687660 27762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:49.689081 27762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:49.690290 27762 hybrid_clock.cc:648] HybridClock initialized: now 1790644369690237 us; error 41 us; skew 500 ppm
I20260929 01:12:49.690680 27762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:49 dist-test-slave-41cl krb5kdc[27749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644369, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:49.693856 27762 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:12:49.695405 27762 webserver.cc:460] Webserver started at http://127.5.198.254:34209/ using document root <none> and password file <none>
I20260929 01:12:49.696079 27762 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:49.696223 27762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:49.696621 27762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:49.699517 27762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "8fad8418558e4bd39df75d813effc1e1"
format_stamp: "Formatted at 2026-09-29 01:12:49 on dist-test-slave-41cl"
server_key: "c2415d039aab231eb0bfed9d5e2b87a0"
server_key_iv: "11e42526b052e616b1debf94c9e43142"
server_key_version: "encryptionkey@0"
I20260929 01:12:49.700263 27762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "8fad8418558e4bd39df75d813effc1e1"
format_stamp: "Formatted at 2026-09-29 01:12:49 on dist-test-slave-41cl"
server_key: "c2415d039aab231eb0bfed9d5e2b87a0"
server_key_iv: "11e42526b052e616b1debf94c9e43142"
server_key_version: "encryptionkey@0"
I20260929 01:12:49.705386 27762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 01:12:49.708884 27777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:49.710239 27762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:12:49.710440 27762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "8fad8418558e4bd39df75d813effc1e1"
format_stamp: "Formatted at 2026-09-29 01:12:49 on dist-test-slave-41cl"
server_key: "c2415d039aab231eb0bfed9d5e2b87a0"
server_key_iv: "11e42526b052e616b1debf94c9e43142"
server_key_version: "encryptionkey@0"
I20260929 01:12:49.710650 27762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:49.754593 27762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:49.758973 27762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:49.759274 27762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:49.778481 27762 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:33191
I20260929 01:12:49.778503 27828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:33191 every 8 connection(s)
I20260929 01:12:49.779841 27762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:49.783491 27829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:49.788039 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 27762
I20260929 01:12:49.788229 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:49.788844 5915 external_mini_cluster.cc:1428] Setting key e86b7729b08109349a95c7b77401ad8a
I20260929 01:12:49.794734 27829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1: Bootstrap starting.
I20260929 01:12:49.797792 27829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:49.798816 27829 log.cc:824] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:49.802150 27829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1: No bootstrap required, opened a new log
I20260929 01:12:49.802229 27832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:49.790685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53321 (local address 127.5.198.254:33191)
0929 01:12:49.791523 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:49.791535 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:49.791574 (+ 39us) server_negotiation.cc:407] Connection header received
0929 01:12:49.793494 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:49.793515 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:49.793886 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:49.794242 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:49.794763 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:49.796051 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:49.797476 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:49.798192 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:49.798390 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:49.799704 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:49.799712 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:49.799726 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:49.800329 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:49.800419 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:49.800456 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:49.800642 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:49.800890 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 01:12:49.801052 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":349,"threads_started":1}
I20260929 01:12:49.806046 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:49.812135 27829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fad8418558e4bd39df75d813effc1e1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33191 } }
I20260929 01:12:49.812628 27829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:49.812752 27829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fad8418558e4bd39df75d813effc1e1, State: Initialized, Role: FOLLOWER
I20260929 01:12:49.813374 27829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fad8418558e4bd39df75d813effc1e1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33191 } }
I20260929 01:12:49.813668 27829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:49.813892 27829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:49.814067 27829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:49.816893 27829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fad8418558e4bd39df75d813effc1e1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33191 } }
I20260929 01:12:49.817466 27829 leader_election.cc:302] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fad8418558e4bd39df75d813effc1e1; no voters:
I20260929 01:12:49.817894 27832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:49.810980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41905 (local address 127.5.198.254:33191)
0929 01:12:49.811189 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:49.811195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:49.811750 (+ 555us) server_negotiation.cc:407] Connection header received
0929 01:12:49.811844 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:49.811852 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:49.811925 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:49.812029 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:49.812545 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:49.813659 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:49.815233 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:49.815937 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:49.816082 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:49.817137 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:49.817143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:49.817148 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:49.817169 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:49.817236 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:49.817257 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:49.817374 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:49.817535 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 01:12:49.817583 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:12:49.818154 27829 leader_election.cc:288] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:49.818311 27839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:49.819994 27839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8fad8418558e4bd39df75d813effc1e1, State: Running, Role: LEADER
I20260929 01:12:49.820358 27829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:49.820621 27839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fad8418558e4bd39df75d813effc1e1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33191 } }
I20260929 01:12:49.824574 27841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fad8418558e4bd39df75d813effc1e1. Latest consensus state: current_term: 1 leader_uuid: "8fad8418558e4bd39df75d813effc1e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fad8418558e4bd39df75d813effc1e1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33191 } } }
I20260929 01:12:49.824610 27840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fad8418558e4bd39df75d813effc1e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fad8418558e4bd39df75d813effc1e1" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 33191 } } }
I20260929 01:12:49.825078 27840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:49.825088 27841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:49.826639 27848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:49.832407 27848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:49.841049 27848 catalog_manager.cc:1347] Generated new cluster ID: 54d7f5bfec3947459d1d4e39732ea81f
I20260929 01:12:49.841167 27848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:49.858021 27848 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:49.860054 27848 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:49.872547 27848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fad8418558e4bd39df75d813effc1e1: Generated new TSK 0
I20260929 01:12:49.873535 27848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:49.887705 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 27762
W20260929 01:12:49.889294 27824 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53321
W20260929 01:12:49.890070 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 27762: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:12:49.890223 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 27762
W20260929 01:12:49.907382 27747 connection.cc:535] client connection to 127.5.198.254:33191 recv error: Network error: failed to read from TLS socket (remote: 127.5.198.254:33191): Connection reset by peer (error 104)
2026-09-29T01:12:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1935 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:49 dist-test-slave-41cl krb5kdc[27860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:49 dist-test-slave-41cl krb5kdc[27860](info): set up 2 sockets
Sep 29 01:12:49 dist-test-slave-41cl krb5kdc[27860](info): commencing operation
krb5kdc: starting...
W20260929 01:12:51.001078 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:51 dist-test-slave-41cl krb5kdc[27860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:12:51.251127 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:40507
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:40507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:51.577487 27873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:51.577759 27873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:51.577853 27873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:51.585737 27873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:51.585880 27873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:51.585934 27873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:51.586007 27873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:51.586056 27873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:51.597020 27873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:40507
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:40507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:51.598232 27873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:51.599449 27873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:51.611110 27879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:51.611374 27881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:51.611485 27878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:51.617933 27873 server_base.cc:1034] running on GCE node
I20260929 01:12:51.618984 27873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:51.621994 27873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:51.623864 27873 hybrid_clock.cc:648] HybridClock initialized: now 1790644371623746 us; error 103 us; skew 500 ppm
I20260929 01:12:51.624735 27873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:51 dist-test-slave-41cl krb5kdc[27860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644371, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:51.635233 27873 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:12:51.637022 27873 webserver.cc:460] Webserver started at http://127.5.198.254:35339/ using document root <none> and password file <none>
I20260929 01:12:51.637727 27873 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:51.637831 27873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:51.638093 27873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:51.641147 27873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "1c04383dc7c842918326bf15b8b2966c"
format_stamp: "Formatted at 2026-09-29 01:12:51 on dist-test-slave-41cl"
server_key: "73f3618f18d240ce5b729bdb5556db98"
server_key_iv: "6e49a8f2c08b23356fe2967cf7e9718e"
server_key_version: "encryptionkey@0"
I20260929 01:12:51.641862 27873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "1c04383dc7c842918326bf15b8b2966c"
format_stamp: "Formatted at 2026-09-29 01:12:51 on dist-test-slave-41cl"
server_key: "73f3618f18d240ce5b729bdb5556db98"
server_key_iv: "6e49a8f2c08b23356fe2967cf7e9718e"
server_key_version: "encryptionkey@0"
I20260929 01:12:51.647528 27873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:12:51.651139 27888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:51.652494 27873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:12:51.652633 27873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "1c04383dc7c842918326bf15b8b2966c"
format_stamp: "Formatted at 2026-09-29 01:12:51 on dist-test-slave-41cl"
server_key: "73f3618f18d240ce5b729bdb5556db98"
server_key_iv: "6e49a8f2c08b23356fe2967cf7e9718e"
server_key_version: "encryptionkey@0"
I20260929 01:12:51.652796 27873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:51.658762 27873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:51.662583 27873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:51.662818 27873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:51.681821 27873 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:40507
I20260929 01:12:51.681854 27939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:40507 every 8 connection(s)
I20260929 01:12:51.683274 27873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:51.685356 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 27873
I20260929 01:12:51.685544 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:51.686092 5915 external_mini_cluster.cc:1428] Setting key 59d94ba532f86ae47158b1f17f7cf1b2
I20260929 01:12:51.687847 27940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:51.693761 27943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:51.687798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37017 (local address 127.5.198.254:40507)
0929 01:12:51.688646 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:51.688657 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:51.688697 (+ 40us) server_negotiation.cc:407] Connection header received
0929 01:12:51.690304 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:51.690324 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:51.690674 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:51.691032 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:51.692168 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:51.692174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:51.692186 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:51.692200 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:51.692271 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:51.692289 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:51.692455 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:51.692620 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 01:12:51.692825 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":376,"threads_started":1}
I20260929 01:12:51.697273 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:51.700708 27940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c: Bootstrap starting.
I20260929 01:12:51.704778 27940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:51.705039 27943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:51.702582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35389 (local address 127.5.198.254:40507)
0929 01:12:51.702736 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:51.702740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:51.703084 (+ 344us) server_negotiation.cc:407] Connection header received
0929 01:12:51.703236 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:51.703240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:51.703287 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:51.703360 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:51.704264 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:51.704269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:51.704273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:51.704294 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:51.704330 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:51.704343 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:51.704426 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:51.704621 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 01:12:51.704669 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 01:12:51.705968 27940 log.cc:824] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:51.708530 27940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c: No bootstrap required, opened a new log
I20260929 01:12:51.715674 27940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c04383dc7c842918326bf15b8b2966c" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40507 } }
I20260929 01:12:51.716046 27940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:51.716180 27940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c04383dc7c842918326bf15b8b2966c, State: Initialized, Role: FOLLOWER
I20260929 01:12:51.716827 27940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c04383dc7c842918326bf15b8b2966c" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40507 } }
I20260929 01:12:51.717048 27940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:51.717180 27940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:51.717365 27940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:51.719400 27940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c04383dc7c842918326bf15b8b2966c" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40507 } }
I20260929 01:12:51.719877 27940 leader_election.cc:302] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c04383dc7c842918326bf15b8b2966c; no voters:
I20260929 01:12:51.720423 27940 leader_election.cc:288] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:51.720588 27950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:51.721310 27950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [term 1 LEADER]: Becoming Leader. State: Replica: 1c04383dc7c842918326bf15b8b2966c, State: Running, Role: LEADER
I20260929 01:12:51.721871 27950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c04383dc7c842918326bf15b8b2966c" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40507 } }
I20260929 01:12:51.722610 27940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:51.724872 27952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c04383dc7c842918326bf15b8b2966c. Latest consensus state: current_term: 1 leader_uuid: "1c04383dc7c842918326bf15b8b2966c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c04383dc7c842918326bf15b8b2966c" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40507 } } }
I20260929 01:12:51.725348 27952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:51.726244 27951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c04383dc7c842918326bf15b8b2966c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c04383dc7c842918326bf15b8b2966c" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40507 } } }
I20260929 01:12:51.726670 27951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:51.728209 27958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:51.733439 27958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:51.742291 27958 catalog_manager.cc:1347] Generated new cluster ID: d4e47717ab7544d6964a21c08711ac54
I20260929 01:12:51.742441 27958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:51.765041 27958 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:51.765444 27958 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:51.778234 27958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c04383dc7c842918326bf15b8b2966c: Generated new TSK 0
I20260929 01:12:51.779062 27958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:51.793089 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 27873
W20260929 01:12:51.794076 27935 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37017
W20260929 01:12:51.794723 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 27873: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:12:51.794858 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 27873
2026-09-29T01:12:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1904 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:51 dist-test-slave-41cl krb5kdc[27971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:51 dist-test-slave-41cl krb5kdc[27971](info): set up 2 sockets
Sep 29 01:12:51 dist-test-slave-41cl krb5kdc[27971](info): commencing operation
krb5kdc: starting...
W20260929 01:12:52.901108 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:53 dist-test-slave-41cl krb5kdc[27971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:12:53.144191 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:32875
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:32875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:53.494004 27984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:53.494421 27984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:53.494597 27984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:53.502799 27984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:53.502898 27984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:53.502952 27984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:53.503012 27984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:53.503055 27984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:53.512660 27984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:32875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:32875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:53.513808 27984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:53.515076 27984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:53.526587 27989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:53.526649 27992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:53.526690 27990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:53.527683 27984 server_base.cc:1034] running on GCE node
I20260929 01:12:53.528445 27984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:53.529938 27984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:53.531165 27984 hybrid_clock.cc:648] HybridClock initialized: now 1790644373531114 us; error 44 us; skew 500 ppm
I20260929 01:12:53.531510 27984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:53 dist-test-slave-41cl krb5kdc[27971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644373, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:53.534592 27984 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:12:53.536087 27984 webserver.cc:460] Webserver started at http://127.5.198.254:46323/ using document root <none> and password file <none>
I20260929 01:12:53.536864 27984 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:53.537009 27984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:53.537307 27984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:53.540189 27984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "f915e1b2b10a4a7d9f4aaf21a09490cc"
format_stamp: "Formatted at 2026-09-29 01:12:53 on dist-test-slave-41cl"
server_key: "e9eb99a1425a8f31d9a8d5b75b54d075"
server_key_iv: "0775320693c0f5f27015871cd8fcc68a"
server_key_version: "encryptionkey@0"
I20260929 01:12:53.540954 27984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "f915e1b2b10a4a7d9f4aaf21a09490cc"
format_stamp: "Formatted at 2026-09-29 01:12:53 on dist-test-slave-41cl"
server_key: "e9eb99a1425a8f31d9a8d5b75b54d075"
server_key_iv: "0775320693c0f5f27015871cd8fcc68a"
server_key_version: "encryptionkey@0"
I20260929 01:12:53.545781 27984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:12:53.549098 27999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:53.550318 27984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:12:53.550511 27984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "f915e1b2b10a4a7d9f4aaf21a09490cc"
format_stamp: "Formatted at 2026-09-29 01:12:53 on dist-test-slave-41cl"
server_key: "e9eb99a1425a8f31d9a8d5b75b54d075"
server_key_iv: "0775320693c0f5f27015871cd8fcc68a"
server_key_version: "encryptionkey@0"
I20260929 01:12:53.550696 27984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:53.566637 27984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:53.570474 27984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:53.570758 27984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:53.588064 27984 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:32875
I20260929 01:12:53.588083 28050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:32875 every 8 connection(s)
I20260929 01:12:53.589334 27984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:53.591741 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 27984
I20260929 01:12:53.591903 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:53.592346 5915 external_mini_cluster.cc:1428] Setting key c3c1b38b6870a51bf382ff9d717efa5f
I20260929 01:12:53.593156 28051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 01:12:53 dist-test-slave-41cl krb5kdc[27971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:12:53.606536 28051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc: Bootstrap starting.
I20260929 01:12:53.610543 28051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:53.611752 28051 log.cc:824] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:53.614526 28051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc: No bootstrap required, opened a new log
I20260929 01:12:53.614990 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:53.594137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44703 (local address 127.5.198.254:32875)
0929 01:12:53.594861 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:53.594877 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:53.594936 (+ 59us) server_negotiation.cc:407] Connection header received
0929 01:12:53.596631 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:53.596656 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:53.597152 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:53.597554 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:53.598759 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:53.600778 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:53.601991 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:53.602988 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:53.603172 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:53.607017 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:53.607033 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:53.607051 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:53.607096 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:53.610898 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:53.611583 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:53.611592 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:53.611599 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:53.611814 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:53.612225 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:53.612233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:53.612237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:53.612868 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:53.613180 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:53.613628 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 01:12:53.614026 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":303,"threads_started":1}
I20260929 01:12:53.619146 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:53.624153 28051 raft_consensus.cc:348] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f915e1b2b10a4a7d9f4aaf21a09490cc" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 32875 } }
I20260929 01:12:53.624725 28051 raft_consensus.cc:374] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:53.624859 28051 raft_consensus.cc:729] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f915e1b2b10a4a7d9f4aaf21a09490cc, State: Initialized, Role: FOLLOWER
I20260929 01:12:53.625538 28051 consensus_queue.cc:260] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f915e1b2b10a4a7d9f4aaf21a09490cc" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 32875 } }
I20260929 01:12:53.625778 28051 raft_consensus.cc:388] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:53.625934 28051 raft_consensus.cc:482] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:53.626111 28051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:53.629144 28051 raft_consensus.cc:504] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f915e1b2b10a4a7d9f4aaf21a09490cc" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 32875 } }
I20260929 01:12:53.629685 28051 leader_election.cc:302] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f915e1b2b10a4a7d9f4aaf21a09490cc; no voters:
I20260929 01:12:53.630326 28051 leader_election.cc:288] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:53.630541 28061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:53.631966 28061 raft_consensus.cc:686] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [term 1 LEADER]: Becoming Leader. State: Replica: f915e1b2b10a4a7d9f4aaf21a09490cc, State: Running, Role: LEADER
I20260929 01:12:53.632556 28061 consensus_queue.cc:237] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f915e1b2b10a4a7d9f4aaf21a09490cc" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 32875 } }
I20260929 01:12:53.633127 28051 sys_catalog.cc:564] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:53.635177 28063 sys_catalog.cc:455] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f915e1b2b10a4a7d9f4aaf21a09490cc. Latest consensus state: current_term: 1 leader_uuid: "f915e1b2b10a4a7d9f4aaf21a09490cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f915e1b2b10a4a7d9f4aaf21a09490cc" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 32875 } } }
I20260929 01:12:53.635550 28063 sys_catalog.cc:458] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:53.636286 28062 sys_catalog.cc:455] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f915e1b2b10a4a7d9f4aaf21a09490cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f915e1b2b10a4a7d9f4aaf21a09490cc" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 32875 } } }
I20260929 01:12:53.637457 28062 sys_catalog.cc:458] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:53.645821 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:53.624232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47959 (local address 127.5.198.254:32875)
0929 01:12:53.624430 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:53.624436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:53.624817 (+ 381us) server_negotiation.cc:407] Connection header received
0929 01:12:53.625025 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:53.625030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:53.625133 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:53.625270 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:53.626960 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:53.628291 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:53.630113 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:53.631050 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:53.631917 (+ 867us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:53.634568 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:53.634577 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:53.634586 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:53.634632 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:53.640533 (+ 5901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:53.641622 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:53.641630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:53.641637 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:53.641929 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:53.642514 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:53.642522 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:53.642528 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:53.643001 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:53.644876 (+ 1875us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:53.645085 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 01:12:53.645390 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 01:12:53.646639 28066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:53.652236 28066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:53.661399 28066 catalog_manager.cc:1347] Generated new cluster ID: 8f385bb4321e488696dd25092c838001
I20260929 01:12:53.661527 28066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:53.678887 28066 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:53.680137 28066 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:53.690253 28066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f915e1b2b10a4a7d9f4aaf21a09490cc: Generated new TSK 0
I20260929 01:12:53.691009 28066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:53.700914 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 27984
I20260929 01:12:53.702091 28046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:12:53.793871 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 27984
2026-09-29T01:12:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:53 dist-test-slave-41cl krb5kdc[28084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:53 dist-test-slave-41cl krb5kdc[28084](info): set up 2 sockets
Sep 29 01:12:53 dist-test-slave-41cl krb5kdc[28084](info): commencing operation
krb5kdc: starting...
W20260929 01:12:54.908929 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:55 dist-test-slave-41cl krb5kdc[28084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:12:55.150563 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:40591
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:40591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:55.485889 28097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:55.486181 28097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:55.486274 28097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:55.493387 28097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:55.493492 28097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:55.493546 28097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:55.493607 28097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:55.493649 28097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:55.503125 28097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:40591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:40591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:55.504299 28097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:55.505512 28097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:55.516825 28102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:55.516848 28103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:55.516969 28105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:55.518126 28097 server_base.cc:1034] running on GCE node
I20260929 01:12:55.518854 28097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:55.520215 28097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:55.521502 28097 hybrid_clock.cc:648] HybridClock initialized: now 1790644375521425 us; error 83 us; skew 500 ppm
I20260929 01:12:55.521888 28097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:55 dist-test-slave-41cl krb5kdc[28084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644375, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:55.525234 28097 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:12:55.526726 28097 webserver.cc:460] Webserver started at http://127.5.198.254:35161/ using document root <none> and password file <none>
I20260929 01:12:55.527390 28097 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:55.527540 28097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:55.527840 28097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:55.530933 28097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "c5bf8535e5f74374abaedb3a5a331159"
format_stamp: "Formatted at 2026-09-29 01:12:55 on dist-test-slave-41cl"
server_key: "a75b85d2c5d4c93b9ef7ff4e117faa36"
server_key_iv: "0599f94cc6bf0e90ab9e1ed6bfed9858"
server_key_version: "encryptionkey@0"
I20260929 01:12:55.531698 28097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "c5bf8535e5f74374abaedb3a5a331159"
format_stamp: "Formatted at 2026-09-29 01:12:55 on dist-test-slave-41cl"
server_key: "a75b85d2c5d4c93b9ef7ff4e117faa36"
server_key_iv: "0599f94cc6bf0e90ab9e1ed6bfed9858"
server_key_version: "encryptionkey@0"
I20260929 01:12:55.537014 28097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 01:12:55.540373 28112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:55.541774 28097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:12:55.541978 28097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "c5bf8535e5f74374abaedb3a5a331159"
format_stamp: "Formatted at 2026-09-29 01:12:55 on dist-test-slave-41cl"
server_key: "a75b85d2c5d4c93b9ef7ff4e117faa36"
server_key_iv: "0599f94cc6bf0e90ab9e1ed6bfed9858"
server_key_version: "encryptionkey@0"
I20260929 01:12:55.542168 28097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:55.561995 28097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:55.565840 28097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:55.566124 28097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:55.584833 28097 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:40591
I20260929 01:12:55.584861 28163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:40591 every 8 connection(s)
I20260929 01:12:55.586119 28097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:55.589695 28164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:55.592020 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 28097
I20260929 01:12:55.592204 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:55.592696 5915 external_mini_cluster.cc:1428] Setting key 8d71aff8effee311b4ddd5643b55801c
I20260929 01:12:55.602748 28164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159: Bootstrap starting.
I20260929 01:12:55.606598 28164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:55.607563 28167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:55.594523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34081 (local address 127.5.198.254:40591)
0929 01:12:55.595336 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:55.595350 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:55.595400 (+ 50us) server_negotiation.cc:407] Connection header received
0929 01:12:55.597781 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:55.597808 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:55.598287 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:55.598725 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:55.599214 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:55.601265 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:55.602556 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:55.603538 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:55.603696 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:55.604876 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:55.604886 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:55.604908 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:55.605588 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:55.605687 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:55.605719 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:55.605839 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:55.605982 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 01:12:55.606230 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":327,"threads_started":1}
I20260929 01:12:55.607908 28164 log.cc:824] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:55.611100 28164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159: No bootstrap required, opened a new log
I20260929 01:12:55.611272 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:55.620523 28164 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bf8535e5f74374abaedb3a5a331159" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40591 } }
I20260929 01:12:55.621282 28164 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:55.621403 28164 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5bf8535e5f74374abaedb3a5a331159, State: Initialized, Role: FOLLOWER
I20260929 01:12:55.622154 28164 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bf8535e5f74374abaedb3a5a331159" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40591 } }
I20260929 01:12:55.622507 28164 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:55.622670 28164 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:55.622860 28164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:55.624420 28167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:55.615798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40279 (local address 127.5.198.254:40591)
0929 01:12:55.616043 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:55.616054 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:55.616624 (+ 570us) server_negotiation.cc:407] Connection header received
0929 01:12:55.617069 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:55.617073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:55.617147 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:55.617364 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:55.618337 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:55.619378 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:55.621491 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:55.622119 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:55.622235 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:55.623523 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:55.623528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:55.623531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:55.623553 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:55.623593 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:55.623609 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:55.623695 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:55.623962 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 01:12:55.624090 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 01:12:55.625948 28164 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bf8535e5f74374abaedb3a5a331159" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40591 } }
I20260929 01:12:55.626464 28164 leader_election.cc:302] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5bf8535e5f74374abaedb3a5a331159; no voters:
I20260929 01:12:55.627041 28164 leader_election.cc:288] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:55.627206 28174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:55.627869 28174 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [term 1 LEADER]: Becoming Leader. State: Replica: c5bf8535e5f74374abaedb3a5a331159, State: Running, Role: LEADER
I20260929 01:12:55.628592 28174 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bf8535e5f74374abaedb3a5a331159" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40591 } }
I20260929 01:12:55.630396 28174 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5bf8535e5f74374abaedb3a5a331159. Latest consensus state: current_term: 1 leader_uuid: "c5bf8535e5f74374abaedb3a5a331159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bf8535e5f74374abaedb3a5a331159" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40591 } } }
I20260929 01:12:55.630776 28174 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:55.631572 28176 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5bf8535e5f74374abaedb3a5a331159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bf8535e5f74374abaedb3a5a331159" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 40591 } } }
I20260929 01:12:55.631960 28176 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:55.634846 28164 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:55.635037 28178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:55.640300 28178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:55.650282 28178 catalog_manager.cc:1347] Generated new cluster ID: 38f12f04e1694f6ba761f77a99022881
I20260929 01:12:55.650413 28178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:55.664346 28178 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:55.666059 28178 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:55.674023 28178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5bf8535e5f74374abaedb3a5a331159: Generated new TSK 0
I20260929 01:12:55.675096 28178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:55.683183 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 28097
W20260929 01:12:55.684405 28159 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34081
W20260929 01:12:55.685190 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 28097: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:12:55.685345 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 28097
2026-09-29T01:12:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1890 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:55 dist-test-slave-41cl krb5kdc[28195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:55 dist-test-slave-41cl krb5kdc[28195](info): set up 2 sockets
Sep 29 01:12:55 dist-test-slave-41cl krb5kdc[28195](info): commencing operation
krb5kdc: starting...
W20260929 01:12:56.789006 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.009s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:56 dist-test-slave-41cl krb5kdc[28195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:12:57.040813 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:42283
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:42283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:57.357122 28208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:57.357394 28208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:57.357580 28208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:57.364679 28208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:57.364800 28208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:57.364909 28208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:57.365013 28208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:57.365096 28208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:57.374832 28208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:42283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:42283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:57.376089 28208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:57.377379 28208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:57.389057 28216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:57.389007 28214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:57.389127 28213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:57.389950 28208 server_base.cc:1034] running on GCE node
I20260929 01:12:57.390727 28208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:57.392127 28208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:57.393348 28208 hybrid_clock.cc:648] HybridClock initialized: now 1790644377393313 us; error 64 us; skew 500 ppm
I20260929 01:12:57.393697 28208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:57 dist-test-slave-41cl krb5kdc[28195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644377, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:57.401589 28208 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:12:57.403151 28208 webserver.cc:460] Webserver started at http://127.5.198.254:32979/ using document root <none> and password file <none>
I20260929 01:12:57.403811 28208 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:57.403913 28208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:57.404178 28208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:57.407240 28208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "7903572bcf504c998686720bfcbfc4fb"
format_stamp: "Formatted at 2026-09-29 01:12:57 on dist-test-slave-41cl"
server_key: "170b2e6ea18ab478b686fedfb651966f"
server_key_iv: "4e59b81c1aeec45cada4d391efa36a68"
server_key_version: "encryptionkey@0"
I20260929 01:12:57.407923 28208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "7903572bcf504c998686720bfcbfc4fb"
format_stamp: "Formatted at 2026-09-29 01:12:57 on dist-test-slave-41cl"
server_key: "170b2e6ea18ab478b686fedfb651966f"
server_key_iv: "4e59b81c1aeec45cada4d391efa36a68"
server_key_version: "encryptionkey@0"
I20260929 01:12:57.413442 28208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 01:12:57.416781 28223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:57.418149 28208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:12:57.418337 28208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "7903572bcf504c998686720bfcbfc4fb"
format_stamp: "Formatted at 2026-09-29 01:12:57 on dist-test-slave-41cl"
server_key: "170b2e6ea18ab478b686fedfb651966f"
server_key_iv: "4e59b81c1aeec45cada4d391efa36a68"
server_key_version: "encryptionkey@0"
I20260929 01:12:57.418534 28208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:57.424315 28208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:57.428720 28208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:57.428963 28208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:57.448832 28208 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:42283
I20260929 01:12:57.448854 28274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:42283 every 8 connection(s)
I20260929 01:12:57.450354 28208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:57.452967 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 28208
I20260929 01:12:57.453150 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:57.453621 5915 external_mini_cluster.cc:1428] Setting key 3d2104448ba09e529cacd4f59c7bbc45
I20260929 01:12:57.454377 28275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:57.461225 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:57.455291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59939 (local address 127.5.198.254:42283)
0929 01:12:57.456076 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:57.456088 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:57.456126 (+ 38us) server_negotiation.cc:407] Connection header received
0929 01:12:57.457688 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:57.457709 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:57.458040 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:57.458364 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:57.459547 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:57.459553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:57.459565 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:57.459579 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:57.459654 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:57.459673 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:57.459839 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:57.459980 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 01:12:57.460263 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":370,"threads_started":1}
I20260929 01:12:57.464684 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:12:57.469015 28275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb: Bootstrap starting.
I20260929 01:12:57.472536 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:57.469901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36683 (local address 127.5.198.254:42283)
0929 01:12:57.470077 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:57.470086 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:57.470631 (+ 545us) server_negotiation.cc:407] Connection header received
0929 01:12:57.470735 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:57.470740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:57.470800 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:57.470924 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:57.471833 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:57.471839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:57.471844 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:12:57.471864 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:57.471914 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:12:57.471939 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:57.472072 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:57.472124 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 01:12:57.472184 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:12:57.473001 28275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb: Neither blocks nor log segments found. Creating new log.
I20260929 01:12:57.474329 28275 log.cc:824] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb: Log is configured to *not* fsync() on all Append() calls
I20260929 01:12:57.476961 28275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb: No bootstrap required, opened a new log
I20260929 01:12:57.484663 28275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7903572bcf504c998686720bfcbfc4fb" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42283 } }
I20260929 01:12:57.484982 28275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:12:57.485056 28275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7903572bcf504c998686720bfcbfc4fb, State: Initialized, Role: FOLLOWER
I20260929 01:12:57.485569 28275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7903572bcf504c998686720bfcbfc4fb" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42283 } }
I20260929 01:12:57.485754 28275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:12:57.485847 28275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:12:57.485977 28275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:12:57.488066 28275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7903572bcf504c998686720bfcbfc4fb" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42283 } }
I20260929 01:12:57.488561 28275 leader_election.cc:302] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7903572bcf504c998686720bfcbfc4fb; no voters:
I20260929 01:12:57.489282 28275 leader_election.cc:288] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:12:57.489444 28285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:12:57.490154 28285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [term 1 LEADER]: Becoming Leader. State: Replica: 7903572bcf504c998686720bfcbfc4fb, State: Running, Role: LEADER
I20260929 01:12:57.490657 28285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7903572bcf504c998686720bfcbfc4fb" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42283 } }
I20260929 01:12:57.491317 28275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:12:57.493465 28286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7903572bcf504c998686720bfcbfc4fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7903572bcf504c998686720bfcbfc4fb" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42283 } } }
I20260929 01:12:57.493935 28286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:57.497028 28292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:12:57.496968 28287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7903572bcf504c998686720bfcbfc4fb. Latest consensus state: current_term: 1 leader_uuid: "7903572bcf504c998686720bfcbfc4fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7903572bcf504c998686720bfcbfc4fb" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 42283 } } }
I20260929 01:12:57.497334 28287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb [sys.catalog]: This master's current role is: LEADER
I20260929 01:12:57.503688 28292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:12:57.516144 28292 catalog_manager.cc:1347] Generated new cluster ID: 417dab902a104b27b27a11f57e595752
I20260929 01:12:57.516259 28292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:12:57.531692 28292 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:12:57.532203 28292 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:12:57.543536 28292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7903572bcf504c998686720bfcbfc4fb: Generated new TSK 0
I20260929 01:12:57.544457 28292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:12:57.550026 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 28208
W20260929 01:12:57.551342 28270 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59939
W20260929 01:12:57.552007 5915 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 28208: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:12:57.552141 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 28208
2026-09-29T01:12:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1864 ms)
[----------] 18 tests from AuthTokenIssuingTest (34853 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:12:57 dist-test-slave-41cl krb5kdc[28306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:12:57 dist-test-slave-41cl krb5kdc[28306](info): set up 2 sockets
Sep 29 01:12:57 dist-test-slave-41cl krb5kdc[28306](info): commencing operation
krb5kdc: starting...
W20260929 01:12:58.657446 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:12:58 dist-test-slave-41cl krb5kdc[28306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:12:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:12:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:12:58.912436 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:44613
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:44613,127.5.198.253:41691,127.5.198.252:35369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:12:59.236720 28319 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:12:59.237071 28319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:59.237188 28319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:59.237336 28319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:59.244483 28319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:59.244562 28319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:59.244664 28319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:59.244732 28319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:59.244837 28319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:59.254400 28319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:44613,127.5.198.253:41691,127.5.198.252:35369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:44613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:59.255650 28319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:59.256989 28319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:59.268900 28327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:59.269043 28324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:59.269007 28325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:59.269657 28319 server_base.cc:1034] running on GCE node
I20260929 01:12:59.270310 28319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:59.271595 28319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:59.272805 28319 hybrid_clock.cc:648] HybridClock initialized: now 1790644379272745 us; error 39 us; skew 500 ppm
I20260929 01:12:59.273080 28319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:59 dist-test-slave-41cl krb5kdc[28306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644379, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:59.276259 28319 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:12:59.277736 28319 webserver.cc:460] Webserver started at http://127.5.198.254:36719/ using document root <none> and password file <none>
I20260929 01:12:59.278389 28319 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:59.278497 28319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:59.278815 28319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:59.281709 28319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "a31d774c745c47c0a46d63d5015db930"
format_stamp: "Formatted at 2026-09-29 01:12:59 on dist-test-slave-41cl"
server_key: "df7d8f69559dc166be7995af401a488a"
server_key_iv: "cdd3f4278a5c87cdad819f10b2bc73d6"
server_key_version: "encryptionkey@0"
I20260929 01:12:59.282483 28319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "a31d774c745c47c0a46d63d5015db930"
format_stamp: "Formatted at 2026-09-29 01:12:59 on dist-test-slave-41cl"
server_key: "df7d8f69559dc166be7995af401a488a"
server_key_iv: "cdd3f4278a5c87cdad819f10b2bc73d6"
server_key_version: "encryptionkey@0"
I20260929 01:12:59.287802 28319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 01:12:59.291260 28334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:59.292645 28319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:12:59.292817 28319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "a31d774c745c47c0a46d63d5015db930"
format_stamp: "Formatted at 2026-09-29 01:12:59 on dist-test-slave-41cl"
server_key: "df7d8f69559dc166be7995af401a488a"
server_key_iv: "cdd3f4278a5c87cdad819f10b2bc73d6"
server_key_version: "encryptionkey@0"
I20260929 01:12:59.293030 28319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:59.319880 28319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:59.324563 28319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:59.324853 28319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:59.343743 28319 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:44613
I20260929 01:12:59.343751 28385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:44613 every 8 connection(s)
I20260929 01:12:59.345221 28319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:12:59.346791 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 28319
I20260929 01:12:59.347060 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:12:59.347844 5915 external_mini_cluster.cc:1428] Setting key f557a5437fb7eb4c9453bf856a3062a0
I20260929 01:12:59.349988 28386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:59.359869 28386 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.198.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.253@KRBTEST.COM" created.
Sep 29 01:12:59 dist-test-slave-41cl krb5kdc[28306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644379, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:12:59.385838 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.362890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44613 (local address 127.0.0.1:36326)
0929 01:12:59.363711 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.363755 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:59.365799 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:59.366731 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:59.366736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:59.367258 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:59.367594 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:59.367610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:59.369964 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:59.369969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:59.371165 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:59.371181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:59.372655 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:59.372660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:59.372810 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:59.373504 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:12:59.373530 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:59.378425 (+ 4895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:59.381707 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:59.381720 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:59.381733 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:59.382113 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:59.382519 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:59.382527 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:59.382530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:59.382680 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:59.383433 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:59.383445 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:59.383649 (+ 204us) client_negotiation.cc:769] Sending connection context
0929 01:12:59.383916 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 01:12:59.384544 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":141,"thread_start_us":357,"threads_started":1}
I20260929 01:12:59.385871 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.364148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.5.198.254:44613)
0929 01:12:59.364866 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:59.364876 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:59.364926 (+ 50us) server_negotiation.cc:407] Connection header received
0929 01:12:59.366179 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:59.366189 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:59.366495 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:59.366723 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:59.367808 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:59.369753 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:59.371324 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:59.372369 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:59.372599 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:59.378654 (+ 6055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:59.378662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:59.378675 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:59.378707 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:59.381551 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:59.382244 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:59.382252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:59.382256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:59.382413 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:59.382785 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:59.382793 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:59.382796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:59.383278 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:59.383511 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:59.384034 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 01:12:59.384380 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":494,"thread_start_us":345,"threads_started":1}
I20260929 01:12:59.388944 28386 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:12:59.390192 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.389751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:41691 (local address 127.0.0.1:43452)
0929 01:12:59.389967 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.390080 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 01:12:59.390451 28336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.198.253:41691: connect: Connection refused (error 111)
I20260929 01:12:59.392123 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.391647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:41691 (local address 127.0.0.1:43468)
0929 01:12:59.391837 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.391936 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 01:12:59.392350 28386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.253:41691: Network error: Client connection negotiation failed: client connection to 127.5.198.253:41691: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.253.keytab.
Entry for principal kudu/127.5.198.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.253.keytab.
I20260929 01:12:59.408012 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.253:41691
--webserver_interface=127.5.198.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.253.keytab
--principal=kudu/127.5.198.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:44613,127.5.198.253:41691,127.5.198.252:35369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:12:59.441718 28386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } attempt: 1
I20260929 01:12:59.443001 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.442477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:41691 (local address 127.0.0.1:43472)
0929 01:12:59.442696 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.442837 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 01:12:59.444039 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.443655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:41691 (local address 127.0.0.1:43482)
0929 01:12:59.443807 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.443890 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 01:12:59.444284 28386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.253:41691: Network error: Client connection negotiation failed: client connection to 127.5.198.253:41691: connect: Connection refused (error 111)
I20260929 01:12:59.512564 28386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } attempt: 2
I20260929 01:12:59.513870 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.513291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:41691 (local address 127.0.0.1:43498)
0929 01:12:59.513576 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.513697 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 01:12:59.515035 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.514592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:41691 (local address 127.0.0.1:43506)
0929 01:12:59.514789 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.514874 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 01:12:59.515275 28386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.253:41691: Network error: Client connection negotiation failed: client connection to 127.5.198.253:41691: connect: Connection refused (error 111)
I20260929 01:12:59.606591 28386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } attempt: 3
I20260929 01:12:59.607905 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.607358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:41691 (local address 127.0.0.1:43514)
0929 01:12:59.607589 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.607705 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 01:12:59.609061 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.608651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:41691 (local address 127.0.0.1:43516)
0929 01:12:59.608808 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.608898 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 01:12:59.609254 28386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.253:41691: Network error: Client connection negotiation failed: client connection to 127.5.198.253:41691: connect: Connection refused (error 111)
W20260929 01:12:59.739617 28393 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:12:59.739974 28393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:12:59.740049 28393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:12:59.740193 28393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:12:59.747391 28393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:12:59.747471 28393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:12:59.747526 28393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:12:59.747576 28393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:12:59.747628 28393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:12:59.752514 28386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } attempt: 4
I20260929 01:12:59.753808 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.753187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:41691 (local address 127.0.0.1:43526)
0929 01:12:59.753530 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.753637 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260929 01:12:59.754868 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.754477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:41691 (local address 127.0.0.1:43530)
0929 01:12:59.754643 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.754734 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 01:12:59.755141 28386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.253:41691: Network error: Client connection negotiation failed: client connection to 127.5.198.253:41691: connect: Connection refused (error 111)
I20260929 01:12:59.758282 28393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:44613,127.5.198.253:41691,127.5.198.252:35369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.253.keytab
--principal=kudu/127.5.198.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.253:41691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.198.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:12:59.759511 28393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:12:59.760850 28393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:12:59.772814 28399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:59.772917 28398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:12:59.772935 28401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:12:59.773854 28393 server_base.cc:1034] running on GCE node
I20260929 01:12:59.774676 28393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:12:59.776041 28393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:12:59.777326 28393 hybrid_clock.cc:648] HybridClock initialized: now 1790644379777240 us; error 55 us; skew 500 ppm
I20260929 01:12:59.777685 28393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:12:59 dist-test-slave-41cl krb5kdc[28306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644379, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:12:59.780931 28393 init.cc:377] Logged in from keytab as kudu/127.5.198.253@KRBTEST.COM (short username kudu)
I20260929 01:12:59.782596 28393 webserver.cc:460] Webserver started at http://127.5.198.253:33833/ using document root <none> and password file <none>
I20260929 01:12:59.783253 28393 fs_manager.cc:359] Metadata directory not provided
I20260929 01:12:59.783339 28393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:12:59.783593 28393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:12:59.786542 28393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/data/instance:
uuid: "6d15cf719c6f45c58132a97fe3ce887c"
format_stamp: "Formatted at 2026-09-29 01:12:59 on dist-test-slave-41cl"
server_key: "7285181e14d309fa3672059ed3efb9d8"
server_key_iv: "e55a8bdd876bb2d4c98ca1e45c85db85"
server_key_version: "encryptionkey@0"
I20260929 01:12:59.787245 28393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/wal/instance:
uuid: "6d15cf719c6f45c58132a97fe3ce887c"
format_stamp: "Formatted at 2026-09-29 01:12:59 on dist-test-slave-41cl"
server_key: "7285181e14d309fa3672059ed3efb9d8"
server_key_iv: "e55a8bdd876bb2d4c98ca1e45c85db85"
server_key_version: "encryptionkey@0"
I20260929 01:12:59.792234 28393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:12:59.795637 28408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:12:59.797219 28393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:12:59.797416 28393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/wal
uuid: "6d15cf719c6f45c58132a97fe3ce887c"
format_stamp: "Formatted at 2026-09-29 01:12:59 on dist-test-slave-41cl"
server_key: "7285181e14d309fa3672059ed3efb9d8"
server_key_iv: "e55a8bdd876bb2d4c98ca1e45c85db85"
server_key_version: "encryptionkey@0"
I20260929 01:12:59.797637 28393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:12:59.818498 28393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:12:59.822813 28393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:12:59.823102 28393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:12:59.841686 28393 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.253:41691
I20260929 01:12:59.841706 28459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.253:41691 every 8 connection(s)
I20260929 01:12:59.843048 28393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/data/info.pb
I20260929 01:12:59.846689 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:12:59.852278 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 28393
I20260929 01:12:59.852488 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-1/wal/instance
I20260929 01:12:59.852977 5915 external_mini_cluster.cc:1428] Setting key 58af32343ef923d01c582fb4f9c593f2
I20260929 01:12:59.854254 28460 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:12:59 dist-test-slave-41cl krb5kdc[28306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644379, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.253@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:12:59.877256 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.857888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.5.198.254:44613)
0929 01:12:59.858155 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:59.858163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:59.859731 (+ 1568us) server_negotiation.cc:407] Connection header received
0929 01:12:59.862741 (+ 3010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:59.862748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:59.862879 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:59.863008 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:59.864891 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:59.865731 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:59.867261 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:59.867675 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:59.867813 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:59.871499 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:59.871504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:59.871507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:59.871539 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:59.874203 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:59.874957 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:59.874967 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:59.874970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:59.875085 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:59.875499 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:59.875504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:59.875509 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:59.875786 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:59.875957 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:59.876683 (+ 726us) server_negotiation.cc:299] Negotiation successful
0929 01:12:59.876916 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:12:59.878314 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.858414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44613 (local address 127.0.0.1:36332)
0929 01:12:59.859481 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.859567 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:59.862391 (+ 2824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:59.863486 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:59.863502 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:59.864066 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:59.864711 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:59.864731 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:59.865916 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:59.865920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:59.867098 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:59.867110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:59.867916 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:59.867924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:59.868155 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:59.868883 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:12:59.868905 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:59.871259 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:59.874358 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:59.874367 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:59.874383 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:59.874823 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:59.875199 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:59.875204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:59.875208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:59.875382 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:59.875924 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:59.875934 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:59.876276 (+ 342us) client_negotiation.cc:769] Sending connection context
0929 01:12:59.876597 (+ 321us) client_negotiation.cc:240] Negotiation successful
0929 01:12:59.877113 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":381,"threads_started":1}
WARNING: no policy specified for kudu/127.5.198.252@KRBTEST.COM; defaulting to no policy
I20260929 01:12:59.880272 28460 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.252@KRBTEST.COM" created.
Sep 29 01:12:59 dist-test-slave-41cl krb5kdc[28306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644379, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.253@KRBTEST.COM for kudu/127.5.198.253@KRBTEST.COM
I20260929 01:12:59.897707 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.881128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:41691 (local address 127.0.0.1:43544)
0929 01:12:59.881305 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.881325 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:12:59.881431 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:12:59.882552 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:12:59.882556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:12:59.882974 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:12:59.883185 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:59.883194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:59.885351 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:59.885358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:59.886864 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:12:59.886880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:12:59.888115 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:12:59.888124 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:12:59.888425 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:59.889157 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:12:59.889184 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:12:59.891900 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:12:59.895154 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:59.895159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:59.895163 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:59.895534 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:59.895909 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:12:59.895914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:12:59.895918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:12:59.896048 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:12:59.896777 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:12:59.896786 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:12:59.896927 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 01:12:59.897087 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 01:12:59.897394 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":39}
I20260929 01:12:59.898393 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.881453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.5.198.253:41691)
0929 01:12:59.882112 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 01:12:59.882117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:12:59.882142 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:12:59.882183 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:12:59.882188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:12:59.882365 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:12:59.882481 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:12:59.883388 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:59.885192 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:59.887030 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:12:59.887959 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:12:59.888104 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:12:59.892124 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:12:59.892135 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:12:59.892150 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:12:59.892182 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:12:59.894994 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:59.895668 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:59.895677 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:59.895682 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:59.895795 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:12:59.896162 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:12:59.896169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:12:59.896174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:12:59.896656 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:12:59.896875 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 01:12:59.897226 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 01:12:59.898147 (+ 921us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":736,"server-negotiator.queue_time_us":486,"thread_start_us":346,"threads_started":1}
I20260929 01:12:59.900610 28460 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:12:59.901813 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.901308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36442)
0929 01:12:59.901477 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.901618 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 01:12:59.902037 28409 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
I20260929 01:12:59.903620 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.903245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36454)
0929 01:12:59.903383 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.903488 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 01:12:59.903841 28460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:35369: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.252.keytab.
Entry for principal kudu/127.5.198.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.252.keytab.
I20260929 01:12:59.917080 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.252:35369
--webserver_interface=127.5.198.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.252.keytab
--principal=kudu/127.5.198.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:38331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:44613,127.5.198.253:41691,127.5.198.252:35369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:12:59.953127 28460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } attempt: 1
I20260929 01:12:59.954463 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.953859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36468)
0929 01:12:59.954092 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.954243 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 01:12:59.955499 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:12:59.955144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36474)
0929 01:12:59.955295 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 01:12:59.955394 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 01:12:59.955718 28460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:35369: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
I20260929 01:13:00.023962 28460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } attempt: 2
I20260929 01:13:00.025381 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.024812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36488)
0929 01:13:00.025139 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.025229 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260929 01:13:00.026515 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.026159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36496)
0929 01:13:00.026304 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.026380 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 01:13:00.026731 28460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:35369: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
I20260929 01:13:00.054406 28386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } attempt: 5
Sep 29 01:13:00 dist-test-slave-41cl krb5kdc[28306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644379, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for kudu/127.5.198.253@KRBTEST.COM
I20260929 01:13:00.069831 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.055253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:41691 (local address 127.0.0.1:43558)
0929 01:13:00.055491 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.055517 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:13:00.055639 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:13:00.056166 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:13:00.056172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:13:00.056661 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:13:00.056848 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:00.056857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.058241 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.058245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:00.059116 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:00.059136 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.060305 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.060310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:00.060561 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.061212 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:13:00.061230 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:13:00.063279 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:13:00.067458 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:00.067462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:00.067465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:00.067841 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:00.068272 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:00.068275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:00.068277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:00.068421 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:00.069091 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:13:00.069095 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:13:00.069234 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 01:13:00.069369 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 01:13:00.069578 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 01:13:00.070055 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.055413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43558 (local address 127.5.198.253:41691)
0929 01:13:00.055627 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.055633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.055656 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:13:00.055763 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.055768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.055849 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.056022 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.057011 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.058100 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.059292 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.060144 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.060293 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.063473 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:00.063479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:00.063484 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:00.063523 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:00.067307 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.068004 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.068009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.068013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.068159 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.068582 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.068586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.068590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.068959 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:00.069103 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:00.069526 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 01:13:00.069745 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:13:00.071216 28386 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:13:00.072230 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.071838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36512)
0929 01:13:00.072031 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.072105 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 01:13:00.073148 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.072831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36528)
0929 01:13:00.072998 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.073059 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 01:13:00.073366 28386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:35369: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
I20260929 01:13:00.117964 28460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } attempt: 3
I20260929 01:13:00.119078 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.118624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36538)
0929 01:13:00.118822 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.118918 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 01:13:00.120134 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.119727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36552)
0929 01:13:00.119952 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.120032 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260929 01:13:00.120343 28460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:35369: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
I20260929 01:13:00.124565 28386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } attempt: 1
I20260929 01:13:00.125705 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.125112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36568)
0929 01:13:00.125358 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.125481 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 01:13:00.126967 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.126500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36572)
0929 01:13:00.126693 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.126796 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 01:13:00.127213 28386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:35369: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
I20260929 01:13:00.195420 28386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } attempt: 2
I20260929 01:13:00.196709 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.196085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36586)
0929 01:13:00.196333 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.196447 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 01:13:00.197817 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.197426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36592)
0929 01:13:00.197587 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.197688 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 01:13:00.198065 28386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:35369: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
W20260929 01:13:00.245370 28467 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:13:00.245679 28467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:13:00.245752 28467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:13:00.245831 28467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:13:00.252893 28467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:13:00.252975 28467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:13:00.253031 28467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:13:00.253082 28467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:13:00.253141 28467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:13:00.262683 28467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:38331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:44613,127.5.198.253:41691,127.5.198.252:35369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.252.keytab
--principal=kudu/127.5.198.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.252:35369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.198.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:13:00.263552 28460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } attempt: 4
I20260929 01:13:00.264020 28467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:13:00.264864 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.264320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36602)
0929 01:13:00.264607 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.264705 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 01:13:00.265610 28467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:13:00.265913 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.265517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36604)
0929 01:13:00.265672 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.265790 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 01:13:00.266108 28460 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:35369: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
W20260929 01:13:00.277247 28472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:13:00.277294 28473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:13:00.277275 28475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:13:00.278476 28467 server_base.cc:1034] running on GCE node
I20260929 01:13:00.279347 28467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:13:00.280740 28467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:13:00.281948 28467 hybrid_clock.cc:648] HybridClock initialized: now 1790644380281861 us; error 55 us; skew 500 ppm
I20260929 01:13:00.282295 28467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:13:00 dist-test-slave-41cl krb5kdc[28306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644380, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:13:00.285415 28467 init.cc:377] Logged in from keytab as kudu/127.5.198.252@KRBTEST.COM (short username kudu)
I20260929 01:13:00.286774 28467 webserver.cc:460] Webserver started at http://127.5.198.252:44573/ using document root <none> and password file <none>
I20260929 01:13:00.287390 28467 fs_manager.cc:359] Metadata directory not provided
I20260929 01:13:00.287473 28467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:13:00.287726 28467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:13:00.290665 28467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/data/instance:
uuid: "475cf02ff9ef4c1496e2cd35cb97c441"
format_stamp: "Formatted at 2026-09-29 01:13:00 on dist-test-slave-41cl"
server_key: "fe5e6348575fb42af05281e093d79cf0"
server_key_iv: "e5c514259c4e71d412e348a3db34b7eb"
server_key_version: "encryptionkey@0"
I20260929 01:13:00.291368 28467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/wal/instance:
uuid: "475cf02ff9ef4c1496e2cd35cb97c441"
format_stamp: "Formatted at 2026-09-29 01:13:00 on dist-test-slave-41cl"
server_key: "fe5e6348575fb42af05281e093d79cf0"
server_key_iv: "e5c514259c4e71d412e348a3db34b7eb"
server_key_version: "encryptionkey@0"
I20260929 01:13:00.296360 28467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 01:13:00.299896 28482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:13:00.301403 28467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:13:00.301586 28467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/wal
uuid: "475cf02ff9ef4c1496e2cd35cb97c441"
format_stamp: "Formatted at 2026-09-29 01:13:00 on dist-test-slave-41cl"
server_key: "fe5e6348575fb42af05281e093d79cf0"
server_key_iv: "e5c514259c4e71d412e348a3db34b7eb"
server_key_version: "encryptionkey@0"
I20260929 01:13:00.301782 28467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:13:00.304271 28386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } attempt: 3
I20260929 01:13:00.305411 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.304955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36614)
0929 01:13:00.305164 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.305248 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 01:13:00.306484 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.306123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36616)
0929 01:13:00.306300 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.306382 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260929 01:13:00.306804 28386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:35369: Network error: Client connection negotiation failed: client connection to 127.5.198.252:35369: connect: Connection refused (error 111)
I20260929 01:13:00.323109 28467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:13:00.327685 28467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:13:00.327961 28467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:13:00.346650 28467 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.252:35369
I20260929 01:13:00.346699 28533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.252:35369 every 8 connection(s)
I20260929 01:13:00.347934 28467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/data/info.pb
I20260929 01:13:00.351202 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 28467
I20260929 01:13:00.351389 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790644023292677-5915-0/minicluster-data/master-2/wal/instance
I20260929 01:13:00.351824 5915 external_mini_cluster.cc:1428] Setting key d47449627d759e00da78abcab9fdb6da
I20260929 01:13:00.352252 28534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:13:00.360800 28534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:13:00 dist-test-slave-41cl krb5kdc[28306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.252@KRBTEST.COM
Sep 29 01:13:00 dist-test-slave-41cl krb5kdc[28306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644380, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.252@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:13:00.375339 28537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.353646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.5.198.252:35369)
0929 01:13:00.354361 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.354372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.354409 (+ 37us) server_negotiation.cc:407] Connection header received
0929 01:13:00.356781 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.356802 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.357173 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.357596 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.359002 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.360218 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.361568 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.362208 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.362395 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.366891 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:00.366899 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:00.366910 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:00.366946 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:00.370727 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.371473 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.371482 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.371490 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.371704 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.372151 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.372154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.372157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.372763 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:00.373098 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:00.373461 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 01:13:00.373891 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":320,"threads_started":1}
I20260929 01:13:00.379117 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.363269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:44613 (local address 127.0.0.1:36334)
0929 01:13:00.363912 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.363945 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 01:13:00.364119 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:13:00.364529 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:13:00.364535 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:13:00.365124 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:13:00.365436 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:00.365459 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.366550 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.366555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:00.367910 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:00.367923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.369086 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.369095 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:00.369325 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.369960 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:13:00.369977 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:13:00.372401 (+ 2424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:13:00.376208 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:00.376217 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:00.376223 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:00.376843 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:00.377246 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:00.377249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:00.377251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:00.377391 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:00.378030 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:13:00.378037 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:13:00.378209 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 01:13:00.378381 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 01:13:00.378716 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":343,"threads_started":1}
I20260929 01:13:00.379143 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.363239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36334 (local address 127.5.198.254:44613)
0929 01:13:00.363409 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.363413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.364043 (+ 630us) server_negotiation.cc:407] Connection header received
0929 01:13:00.364245 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.364250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.364321 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.364424 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.365602 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.366385 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.368069 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.368822 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.368948 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.372652 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:00.372658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:00.372662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:00.372701 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:00.376089 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.376965 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.376972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.376975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.377132 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.377509 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.377537 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.377540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.377893 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:00.378050 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:00.378520 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 01:13:00.378733 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:13:00.380493 28534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:13:00 dist-test-slave-41cl krb5kdc[28306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
Sep 29 01:13:00 dist-test-slave-41cl krb5kdc[28306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644380, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.252@KRBTEST.COM for kudu/127.5.198.253@KRBTEST.COM
I20260929 01:13:00.395635 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.378974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.5.198.254:44613)
0929 01:13:00.379860 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.379866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.379882 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:13:00.379927 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.379941 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.380006 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.380106 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.381607 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.382737 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.384334 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.385090 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.385237 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.389675 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:00.389684 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:00.389690 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:00.389731 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:00.392773 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.393418 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.393424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.393428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.393594 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.393978 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.393985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.393988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.394340 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:00.394559 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:00.395001 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 01:13:00.395222 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":508,"threads_started":1}
I20260929 01:13:00.395645 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.381238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:41691 (local address 127.0.0.1:43560)
0929 01:13:00.381550 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.381575 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:13:00.381687 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:13:00.381991 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:13:00.381996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:13:00.382419 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:13:00.382610 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:00.382620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.383973 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.383977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:00.385364 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:00.385381 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.386371 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.386375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:00.386519 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.387099 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:13:00.387116 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:13:00.389768 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:13:00.393020 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:00.393024 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:00.393026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:00.393443 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:00.393873 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:00.393876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:00.393878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:00.394089 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:00.394681 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:13:00.394684 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:13:00.394847 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 01:13:00.395014 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 01:13:00.395268 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 01:13:00.395730 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.381346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.5.198.253:41691)
0929 01:13:00.381510 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.381514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.381660 (+ 146us) server_negotiation.cc:407] Connection header received
0929 01:13:00.381803 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.381807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.381864 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.381937 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.382748 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.383813 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.385541 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.386189 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.386316 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.389950 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:00.389955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:00.389958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:00.389987 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:00.392887 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.393572 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.393576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.393579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.393747 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.394214 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.394218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.394220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.394548 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:00.394697 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:00.395126 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 01:13:00.395440 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 01:13:00.396903 28534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:13:00 dist-test-slave-41cl krb5kdc[28306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644380, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.252@KRBTEST.COM for kudu/127.5.198.252@KRBTEST.COM
Sep 29 01:13:00 dist-test-slave-41cl krb5kdc[28306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.253@KRBTEST.COM
I20260929 01:13:00.411899 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.397611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36630)
0929 01:13:00.397868 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.397889 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:13:00.397986 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:13:00.398365 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:13:00.398370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:13:00.398842 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:13:00.399049 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:00.399066 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.400042 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.400046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:00.401254 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:00.401271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.402058 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.402067 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:00.402296 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.402875 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:13:00.402893 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:13:00.405437 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:13:00.409111 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:00.409116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:00.409120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:00.409604 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:00.410029 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:00.410033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:00.410036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:00.410194 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:00.410934 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:13:00.410939 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:13:00.411097 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 01:13:00.411268 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 01:13:00.411479 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260929 01:13:00.411979 28537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.397788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.5.198.252:35369)
0929 01:13:00.398104 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.398109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.398123 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:13:00.398163 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.398169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.398231 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.398318 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.399200 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.399912 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.401406 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.401880 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.401984 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.405628 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:00.405632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:00.405635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:00.405662 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:00.408965 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.409743 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.409747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.409750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.409905 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.410321 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.410327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.410331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.410727 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:00.410923 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:00.411417 (+ 494us) server_negotiation.cc:299] Negotiation successful
0929 01:13:00.411681 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260929 01:13:00.413496 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.397893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43574 (local address 127.5.198.253:41691)
0929 01:13:00.398050 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.398056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.398072 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:13:00.398120 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.398124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.398188 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.398291 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.399821 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.400633 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.402204 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.402600 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.402734 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.407255 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:00.407260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:00.407263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:00.407292 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:00.410064 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.410941 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.410945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.410949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.411106 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.411560 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.411565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.411569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.411882 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:00.412155 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:00.412954 (+ 799us) server_negotiation.cc:299] Negotiation successful
0929 01:13:00.413103 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:13:00.417290 28534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441: Bootstrap starting.
I20260929 01:13:00.418360 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:13:00.421094 28534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441: Neither blocks nor log segments found. Creating new log.
I20260929 01:13:00.422230 28534 log.cc:824] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441: Log is configured to *not* fsync() on all Append() calls
I20260929 01:13:00.425027 28534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441: No bootstrap required, opened a new log
I20260929 01:13:00.421875 28361 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a31d774c745c47c0a46d63d5015db930"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36336
I20260929 01:13:00.432053 28534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } }
I20260929 01:13:00.432382 28534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475cf02ff9ef4c1496e2cd35cb97c441, State: Initialized, Role: FOLLOWER
I20260929 01:13:00.432895 28534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } }
I20260929 01:13:00.434001 28541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } } }
I20260929 01:13:00.434465 28541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:13:00.434875 28534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:13:00.444288 28552 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:13:00.444458 28552 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:13:00.442451 28467 master_runner.cc:186] Error getting master registration for 127.5.198.254:44613: OK, instance_id { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" instance_seqno: 1790644379338393 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:13:00.453763 28467 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:13:00.484042 28386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } attempt: 4
Sep 29 01:13:00 dist-test-slave-41cl krb5kdc[28306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644379, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for kudu/127.5.198.252@KRBTEST.COM
I20260929 01:13:00.501027 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.484836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36632)
0929 01:13:00.485062 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.485086 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:13:00.485220 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:13:00.485668 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:13:00.485674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:13:00.486206 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:13:00.486425 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:00.486438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.487937 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.487943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:00.489426 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:00.489441 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.490455 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.490462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:00.490678 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.491348 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:13:00.491371 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:13:00.494271 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:13:00.498254 (+ 3983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:00.498259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:00.498263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:00.498751 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:00.499139 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:00.499144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:00.499147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:00.499313 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:00.499985 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:13:00.499989 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:13:00.500161 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 01:13:00.500310 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 01:13:00.500622 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 01:13:00.501075 28537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.485123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.5.198.252:35369)
0929 01:13:00.485321 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.485326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.485349 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:13:00.485421 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.485426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.485500 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.485605 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.486575 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.487785 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.489564 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.490280 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.490413 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.494465 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:00.494473 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:00.494478 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:00.494515 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:00.498131 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.498867 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.498873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.498877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.499023 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.499469 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.499474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.499478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.499865 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:00.500012 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:00.500446 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 01:13:00.500682 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:13:00.507131 28386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930: Bootstrap starting.
I20260929 01:13:00.511068 28386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930: Neither blocks nor log segments found. Creating new log.
I20260929 01:13:00.512311 28386 log.cc:824] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930: Log is configured to *not* fsync() on all Append() calls
I20260929 01:13:00.515669 28386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930: No bootstrap required, opened a new log
I20260929 01:13:00.523275 28386 raft_consensus.cc:348] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } }
I20260929 01:13:00.523685 28386 raft_consensus.cc:729] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a31d774c745c47c0a46d63d5015db930, State: Initialized, Role: FOLLOWER
I20260929 01:13:00.524163 28386 consensus_queue.cc:260] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } }
I20260929 01:13:00.525187 28555 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } } }
I20260929 01:13:00.525581 28555 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:13:00.526400 28386 sys_catalog.cc:564] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:13:00.526465 28361 raft_consensus.cc:482] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 01:13:00.526677 28361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:13:00.529253 28361 raft_consensus.cc:504] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } }
I20260929 01:13:00.530720 28361 leader_election.cc:288] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [CANDIDATE]: Term 1 election: Requested vote from peers 6d15cf719c6f45c58132a97fe3ce887c (127.5.198.253:41691), 475cf02ff9ef4c1496e2cd35cb97c441 (127.5.198.252:35369)
I20260929 01:13:00.531788 28509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a31d774c745c47c0a46d63d5015db930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "475cf02ff9ef4c1496e2cd35cb97c441"
I20260929 01:13:00.532289 28509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:13:00.535844 28509 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a31d774c745c47c0a46d63d5015db930 in term 1.
I20260929 01:13:00.536536 28335 leader_election.cc:302] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 475cf02ff9ef4c1496e2cd35cb97c441, a31d774c745c47c0a46d63d5015db930; no voters:
I20260929 01:13:00.536989 28555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:13:00.537628 28555 raft_consensus.cc:686] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [term 1 LEADER]: Becoming Leader. State: Replica: a31d774c745c47c0a46d63d5015db930, State: Running, Role: LEADER
I20260929 01:13:00.538162 28555 consensus_queue.cc:237] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } }
I20260929 01:13:00.541244 28558 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a31d774c745c47c0a46d63d5015db930. Latest consensus state: current_term: 1 leader_uuid: "a31d774c745c47c0a46d63d5015db930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } } }
I20260929 01:13:00.541625 28558 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [sys.catalog]: This master's current role is: LEADER
I20260929 01:13:00.542832 28561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:13:00.548776 28561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:13:00.561564 28509 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441 [term 1 FOLLOWER]: Refusing update from remote peer a31d774c745c47c0a46d63d5015db930: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:13:00.562537 28558 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:13:00.565291 28460 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } attempt: 5
Sep 29 01:13:00 dist-test-slave-41cl krb5kdc[28306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644379, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.253@KRBTEST.COM for kudu/127.5.198.252@KRBTEST.COM
W20260929 01:13:00.577670 28340 tablet.cc:2369] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:13:00.580139 28488 tablet.cc:2369] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:13:00.585955 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.565818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:35369 (local address 127.0.0.1:36644)
0929 01:13:00.566203 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:00.566228 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:13:00.566371 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:13:00.567571 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:13:00.567578 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:13:00.568205 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:13:00.568511 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:00.568528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.570189 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.570197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:00.571974 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:00.571992 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.573337 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.573343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:00.573577 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.574296 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:13:00.574328 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:13:00.577251 (+ 2923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:13:00.582297 (+ 5046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:00.582306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:00.582313 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:00.583013 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:00.583559 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:00.583567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:00.583574 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:00.583775 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:00.584632 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:13:00.584640 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:13:00.584852 (+ 212us) client_negotiation.cc:769] Sending connection context
0929 01:13:00.585078 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 01:13:00.585403 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234}
I20260929 01:13:00.587188 28537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.566818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.5.198.252:35369)
0929 01:13:00.567037 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.567045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.567067 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:13:00.567129 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.567138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.567230 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.567340 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.568717 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.569995 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.572171 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.573009 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.573142 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.577459 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:00.577467 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:00.577473 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:00.577520 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:00.582060 (+ 4540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.583187 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.583196 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.583202 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.583400 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.583936 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.583944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.583949 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.584373 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:00.584555 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:00.586410 (+ 1855us) server_negotiation.cc:299] Negotiation successful
0929 01:13:00.586715 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:13:00.597402 28460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c: Bootstrap starting.
I20260929 01:13:00.601873 28460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c: Neither blocks nor log segments found. Creating new log.
I20260929 01:13:00.605542 28460 log.cc:824] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:13:00.615082 28460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c: No bootstrap required, opened a new log
I20260929 01:13:00.624459 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.561144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.5.198.253:41691)
0929 01:13:00.561330 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.561338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.592775 (+ 31437us) server_negotiation.cc:407] Connection header received
0929 01:13:00.593156 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.593165 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.593243 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.593312 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.595402 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.596501 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.598294 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.598894 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.599001 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.616679 (+ 17678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:00.616686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:00.616691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:00.616734 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:00.620675 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.621880 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.621886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.621889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.622068 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.622558 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.622563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.622567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.622925 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:00.623177 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:00.623811 (+ 634us) server_negotiation.cc:299] Negotiation successful
0929 01:13:00.624079 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:13:00.637914 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.565282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36338 (local address 127.5.198.254:44613)
0929 01:13:00.565464 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.565469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.602540 (+ 37071us) server_negotiation.cc:407] Connection header received
0929 01:13:00.603050 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.603057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.603150 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.603221 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.606773 (+ 3552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.607922 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.616810 (+ 8888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.617634 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.617773 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.629712 (+ 11939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:00.629719 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:00.629725 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:00.629774 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:00.633556 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.634434 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.634441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.634445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.634618 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.635102 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.635107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.635111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.635476 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:00.635757 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:00.636882 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0929 01:13:00.637345 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:13:00.642889 28541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a31d774c745c47c0a46d63d5015db930. Latest consensus state: current_term: 1 leader_uuid: "a31d774c745c47c0a46d63d5015db930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } } }
I20260929 01:13:00.643254 28541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:13:00.647130 28569 mvcc.cc:205] Tried to move back new op lower bound from 7334479382764523520 to 7334479382688874496. Current Snapshot: MvccSnapshot[applied={T|T < 7334479382764523520}]
I20260929 01:13:00.649927 28582 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:13:00.650731 28555 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a31d774c745c47c0a46d63d5015db930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } } }
I20260929 01:13:00.651084 28555 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [sys.catalog]: This master's current role is: LEADER
I20260929 01:13:00.657678 28537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:00.579434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36650 (local address 127.5.198.252:35369)
0929 01:13:00.587438 (+ 8004us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.587446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.618622 (+ 31176us) server_negotiation.cc:407] Connection header received
0929 01:13:00.618944 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.618952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.619076 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.619158 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.621892 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.623000 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.627410 (+ 4410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:00.639946 (+ 12536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:00.640098 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:00.645886 (+ 5788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:00.645893 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:00.645898 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:00.645941 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:00.650722 (+ 4781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.652666 (+ 1944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.652678 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.652683 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.653012 (+ 329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:00.653850 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:00.653858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:00.653863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:00.654656 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:00.655096 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:00.656153 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0929 01:13:00.656723 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7872,"thread_start_us":11780,"threads_started":1}
I20260929 01:13:00.658237 28541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a31d774c745c47c0a46d63d5015db930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } } }
I20260929 01:13:00.658501 28541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:13:00.654070 28319 master_runner.cc:186] Error getting master registration for 127.5.198.253:41691: OK, instance_id { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" instance_seqno: 1790644379836441 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:13:00.663389 28319 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:13:00.666045 28460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } }
I20260929 01:13:00.666466 28460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d15cf719c6f45c58132a97fe3ce887c, State: Initialized, Role: FOLLOWER
I20260929 01:13:00.666982 28460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } }
I20260929 01:13:00.669698 28561 catalog_manager.cc:1347] Generated new cluster ID: 1c78f6b3a8f5441487b5830daa443d13
I20260929 01:13:00.669798 28561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:13:00.673244 28434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:13:00.674032 28460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:13:00.687064 28434 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c [term 1 FOLLOWER]: Refusing update from remote peer a31d774c745c47c0a46d63d5015db930: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:13:00.673820 28435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a31d774c745c47c0a46d63d5015db930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6d15cf719c6f45c58132a97fe3ce887c"
I20260929 01:13:00.689574 28558 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:13:00.691013 28435 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a31d774c745c47c0a46d63d5015db930 in term 1.
I20260929 01:13:00.670812 28585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } } }
I20260929 01:13:00.691588 28585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:13:00.745820 28585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a31d774c745c47c0a46d63d5015db930. Latest consensus state: current_term: 1 leader_uuid: "a31d774c745c47c0a46d63d5015db930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } } }
I20260929 01:13:00.746272 28585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:13:00.752187 28592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a31d774c745c47c0a46d63d5015db930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } } }
I20260929 01:13:00.752882 28592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:13:00.789635 28555 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a31d774c745c47c0a46d63d5015db930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31d774c745c47c0a46d63d5015db930" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 44613 } } peers { permanent_uuid: "6d15cf719c6f45c58132a97fe3ce887c" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 41691 } } peers { permanent_uuid: "475cf02ff9ef4c1496e2cd35cb97c441" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 35369 } } }
I20260929 01:13:00.789942 28555 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930 [sys.catalog]: This master's current role is: LEADER
I20260929 01:13:00.798017 28561 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:13:00.800824 28561 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:13:00.833909 28561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a31d774c745c47c0a46d63d5015db930: Generated new TSK 0
I20260929 01:13:00.835260 28561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:13:00.841717 28600 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 01:13:00.928323 28464 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:00.914745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.5.198.253:41691)
0929 01:13:00.914974 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.914984 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.925791 (+ 10807us) server_negotiation.cc:407] Connection header received
0929 01:13:00.926137 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.926146 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.926266 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.926350 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.928085 (+ 1735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43578: BlockingRecv error: recv got EOF from 127.0.0.1:43578 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 01:13:00.928339 28390 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:00.915129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36352 (local address 127.5.198.254:44613)
0929 01:13:00.915346 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.915354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.925469 (+ 10115us) server_negotiation.cc:407] Connection header received
0929 01:13:00.926629 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.926636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.926727 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.926803 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.928071 (+ 1268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36352: BlockingRecv error: recv got EOF from 127.0.0.1:36352 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 01:13:00.930733 28537 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:00.926011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.5.198.252:35369)
0929 01:13:00.926176 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.926182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.929273 (+ 3091us) server_negotiation.cc:407] Connection header received
0929 01:13:00.929442 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.929448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.929550 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.929695 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.930481 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36654: BlockingRecv error: recv got EOF from 127.0.0.1:36654 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 01:13:00.938344 28464 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:00.936225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43580 (local address 127.5.198.253:41691)
0929 01:13:00.936376 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.936382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.936905 (+ 523us) server_negotiation.cc:407] Connection header received
0929 01:13:00.937080 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.937086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.937221 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.937330 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.938046 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43580: BlockingRecv error: recv got EOF from 127.0.0.1:43580 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 01:13:00.946334 28464 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:00.942855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43582 (local address 127.5.198.253:41691)
0929 01:13:00.943034 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.943039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.944975 (+ 1936us) server_negotiation.cc:407] Connection header received
0929 01:13:00.945159 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.945167 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.945255 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.945361 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.946142 (+ 781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43582: BlockingRecv error: recv got EOF from 127.0.0.1:43582 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 01:13:00.955868 28464 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:00.953715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43590 (local address 127.5.198.253:41691)
0929 01:13:00.953896 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.953904 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.954418 (+ 514us) server_negotiation.cc:407] Connection header received
0929 01:13:00.954610 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.954617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.954725 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.954837 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.955632 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43590: BlockingRecv error: recv got EOF from 127.0.0.1:43590 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 01:13:00.968258 28464 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:00.965976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43596 (local address 127.5.198.253:41691)
0929 01:13:00.966198 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.966206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.966711 (+ 505us) server_negotiation.cc:407] Connection header received
0929 01:13:00.966907 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.966914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.967015 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.967143 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.968004 (+ 861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43596: BlockingRecv error: recv got EOF from 127.0.0.1:43596 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260929 01:13:00.984789 28464 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:00.982463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43600 (local address 127.5.198.253:41691)
0929 01:13:00.982642 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:00.982649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:00.983188 (+ 539us) server_negotiation.cc:407] Connection header received
0929 01:13:00.983394 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:00.983402 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:00.983506 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:00.983630 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:00.984546 (+ 916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43600: BlockingRecv error: recv got EOF from 127.0.0.1:43600 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 01:13:01.006837 28464 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:01.004441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.5.198.253:41691)
0929 01:13:01.004659 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:01.004667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:01.005219 (+ 552us) server_negotiation.cc:407] Connection header received
0929 01:13:01.005433 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:01.005439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:01.005546 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:01.005656 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:01.006561 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43608: BlockingRecv error: recv got EOF from 127.0.0.1:43608 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260929 01:13:01.044981 28464 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:01.042810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43610 (local address 127.5.198.253:41691)
0929 01:13:01.043085 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:01.043091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:01.043403 (+ 312us) server_negotiation.cc:407] Connection header received
0929 01:13:01.043628 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:01.043633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:01.043721 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:01.043817 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:01.044727 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43610: BlockingRecv error: recv got EOF from 127.0.0.1:43610 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
W20260929 01:13:01.114805 28464 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:01.112588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43618 (local address 127.5.198.253:41691)
0929 01:13:01.112796 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:01.112803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:01.113263 (+ 460us) server_negotiation.cc:407] Connection header received
0929 01:13:01.113547 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:01.113552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:01.113632 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:01.113752 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:01.114550 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43618: BlockingRecv error: recv got EOF from 127.0.0.1:43618 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 01:13:01.249912 28464 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:01.248225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.5.198.253:41691)
0929 01:13:01.248410 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:01.248414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:01.248715 (+ 301us) server_negotiation.cc:407] Connection header received
0929 01:13:01.248881 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:01.248886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:01.248968 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:01.249069 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:01.249710 (+ 641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43634: BlockingRecv error: recv got EOF from 127.0.0.1:43634 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:13:01.446439 28552 catalog_manager.cc:1259] Loaded cluster ID: 1c78f6b3a8f5441487b5830daa443d13
I20260929 01:13:01.446543 28552 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441: loading cluster ID for follower catalog manager: success
I20260929 01:13:01.449046 28552 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441: acquiring CA information for follower catalog manager: success
I20260929 01:13:01.450775 28552 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 475cf02ff9ef4c1496e2cd35cb97c441: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:13:01.510885 28464 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:01.509306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.5.198.253:41691)
0929 01:13:01.509510 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:01.509515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:01.509713 (+ 198us) server_negotiation.cc:407] Connection header received
0929 01:13:01.509876 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:01.509880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:01.509956 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:01.510060 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:01.510720 (+ 660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43644: BlockingRecv error: recv got EOF from 127.0.0.1:43644 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 01:13:01.846628 28600 catalog_manager.cc:1259] Loaded cluster ID: 1c78f6b3a8f5441487b5830daa443d13
I20260929 01:13:01.846772 28600 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c: loading cluster ID for follower catalog manager: success
I20260929 01:13:01.850569 28600 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c: acquiring CA information for follower catalog manager: success
I20260929 01:13:01.853042 28600 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6d15cf719c6f45c58132a97fe3ce887c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:13:02.035735 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:02.028614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.5.198.253:41691)
0929 01:13:02.029456 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:02.029460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:02.029478 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:13:02.029541 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:02.029545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:02.029687 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:02.029791 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:13:02.030345 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:02.031185 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:02.033125 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:02.033605 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:02.033713 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:02.034323 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:13:02.034948 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:13:02.035096 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:02.035291 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 01:13:02.035396 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":572,"threads_started":1}
I20260929 01:13:02.060868 28672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:02.042322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36670 (local address 127.5.198.252:35369)
0929 01:13:02.054589 (+ 12267us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:02.054596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:02.054620 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:13:02.054743 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:02.054749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:02.054965 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:02.055165 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:13:02.055609 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:02.056802 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:02.058142 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:02.058779 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:02.058913 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:02.059475 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:13:02.060136 (+ 661us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:13:02.060289 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:02.060402 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 01:13:02.060526 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12101,"thread_start_us":11990,"threads_started":1}
I20260929 01:13:02.089282 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:02.067904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43666 (local address 127.5.198.253:41691)
0929 01:13:02.068140 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:02.068149 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:02.081449 (+ 13300us) server_negotiation.cc:407] Connection header received
0929 01:13:02.081721 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:02.081726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:02.081825 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:02.081891 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:13:02.083181 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:02.084288 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:02.086917 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:02.087564 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:02.087683 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:02.088360 (+ 677us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:13:02.088517 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:13:02.088582 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:02.088812 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 01:13:02.088928 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 01:13:02.091235 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:02.067439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36358 (local address 127.5.198.254:44613)
0929 01:13:02.080295 (+ 12856us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:02.080305 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:02.081371 (+ 1066us) server_negotiation.cc:407] Connection header received
0929 01:13:02.081512 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:02.081519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:02.081727 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:02.081912 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:13:02.082947 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:02.084599 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:02.088202 (+ 3603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:02.088865 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:02.088972 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:02.089877 (+ 905us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:13:02.090292 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:13:02.090425 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:02.090875 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 01:13:02.090956 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12683,"thread_start_us":13038,"threads_started":1}
I20260929 01:13:02.093715 28672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:02.081613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.5.198.252:35369)
0929 01:13:02.081780 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:02.081786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:02.087396 (+ 5610us) server_negotiation.cc:407] Connection header received
0929 01:13:02.087682 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:02.087688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:02.087779 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:02.087850 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:13:02.088736 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:02.089832 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:02.091838 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:02.092260 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:02.092409 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:02.092945 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:13:02.093093 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:13:02.093199 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:02.093363 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 01:13:02.093452 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:13:02.096757 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 28319
I20260929 01:13:02.098067 28381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:13:02.229787 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 28319
I20260929 01:13:02.253862 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 28393
I20260929 01:13:02.255033 28455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:13:02.362613 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 28393
I20260929 01:13:02.382881 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 28467
I20260929 01:13:02.384050 28529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:13:02.493176 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 28467
2026-09-29T01:13:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4948 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:13:02 dist-test-slave-41cl krb5kdc[28686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:13:02 dist-test-slave-41cl krb5kdc[28686](info): set up 2 sockets
Sep 29 01:13:02 dist-test-slave-41cl krb5kdc[28686](info): commencing operation
krb5kdc: starting...
W20260929 01:13:03.641119 5915 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/test-user.keytab.
Sep 29 01:13:03 dist-test-slave-41cl krb5kdc[28686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:13:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:13:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
Entry for principal kudu/127.5.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab.
I20260929 01:13:03.899634 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:34893
--webserver_interface=127.5.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:34893,127.5.198.253:43107,127.5.198.252:38157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:13:04.223929 28699 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:13:04.224325 28699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:13:04.224458 28699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:13:04.224624 28699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:13:04.231797 28699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:13:04.231940 28699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:13:04.232059 28699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:13:04.232148 28699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:13:04.232237 28699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:13:04.242309 28699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:34893,127.5.198.253:43107,127.5.198.252:38157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.254.keytab
--principal=kudu/127.5.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.254:34893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:13:04.243650 28699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:13:04.244999 28699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:13:04.257007 28704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:13:04.256996 28705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:13:04.257577 28707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:13:04.257908 28699 server_base.cc:1034] running on GCE node
I20260929 01:13:04.258718 28699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:13:04.260243 28699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:13:04.261485 28699 hybrid_clock.cc:648] HybridClock initialized: now 1790644384261396 us; error 89 us; skew 500 ppm
I20260929 01:13:04.261832 28699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:13:04 dist-test-slave-41cl krb5kdc[28686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644384, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:13:04.265163 28699 init.cc:377] Logged in from keytab as kudu/127.5.198.254@KRBTEST.COM (short username kudu)
I20260929 01:13:04.266673 28699 webserver.cc:460] Webserver started at http://127.5.198.254:46301/ using document root <none> and password file <none>
I20260929 01:13:04.267362 28699 fs_manager.cc:359] Metadata directory not provided
I20260929 01:13:04.267496 28699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:13:04.267820 28699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:13:04.270887 28699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/data/instance:
uuid: "9edb94110ac644a8b638f314696a91da"
format_stamp: "Formatted at 2026-09-29 01:13:04 on dist-test-slave-41cl"
server_key: "5c273a799cebf5328ab5cae0f9ee16a1"
server_key_iv: "c54b4be85b34f9b9f099b4baf66d7ac0"
server_key_version: "encryptionkey@0"
I20260929 01:13:04.271684 28699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/wal/instance:
uuid: "9edb94110ac644a8b638f314696a91da"
format_stamp: "Formatted at 2026-09-29 01:13:04 on dist-test-slave-41cl"
server_key: "5c273a799cebf5328ab5cae0f9ee16a1"
server_key_iv: "c54b4be85b34f9b9f099b4baf66d7ac0"
server_key_version: "encryptionkey@0"
I20260929 01:13:04.276842 28699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260929 01:13:04.280234 28714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:13:04.281955 28699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:13:04.282174 28699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/wal
uuid: "9edb94110ac644a8b638f314696a91da"
format_stamp: "Formatted at 2026-09-29 01:13:04 on dist-test-slave-41cl"
server_key: "5c273a799cebf5328ab5cae0f9ee16a1"
server_key_iv: "c54b4be85b34f9b9f099b4baf66d7ac0"
server_key_version: "encryptionkey@0"
I20260929 01:13:04.282416 28699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:13:04.310402 28699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:13:04.314819 28699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:13:04.315110 28699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:13:04.334167 28699 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.254:34893
I20260929 01:13:04.334187 28765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.254:34893 every 8 connection(s)
I20260929 01:13:04.335574 28699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/data/info.pb
I20260929 01:13:04.339349 28766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:13:04.344907 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 28699
I20260929 01:13:04.345091 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-0/wal/instance
I20260929 01:13:04.345553 5915 external_mini_cluster.cc:1428] Setting key 760d1053b6c1df18a09fe0cad3c43c8b
I20260929 01:13:04.345988 28766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:13:04 dist-test-slave-41cl krb5kdc[28686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644384, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.198.253@KRBTEST.COM; defaulting to no policy
I20260929 01:13:04.369622 28769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.349128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.5.198.254:34893)
0929 01:13:04.351078 (+ 1950us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:04.351084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:04.351128 (+ 44us) server_negotiation.cc:407] Connection header received
0929 01:13:04.353405 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:04.353410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:04.353658 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:04.353833 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:04.355201 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:04.356603 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:04.357968 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:04.358522 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:04.358718 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:04.362479 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:04.362487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:04.362499 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:04.362531 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:04.365327 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:04.366008 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:04.366014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:04.366016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:04.366155 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:04.366544 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:04.366552 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:04.366555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:04.367051 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:04.367263 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:04.367780 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 01:13:04.368110 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1771,"spinlock_wait_cycles":30976,"thread_start_us":1469,"threads_started":1}
I20260929 01:13:04.369663 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.349128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:34893 (local address 127.0.0.1:33214)
0929 01:13:04.350075 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.350132 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 01:13:04.353005 (+ 2873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:13:04.353919 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:13:04.353933 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:13:04.354620 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:13:04.354977 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:04.354994 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:04.356749 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:04.356756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:04.357823 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:04.357835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:04.358754 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:04.358761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:04.358936 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:04.359645 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:13:04.359676 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:13:04.362248 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:13:04.365467 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:04.365480 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:04.365492 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:04.365863 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:04.366274 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:04.366278 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:04.366282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:04.366440 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:04.367196 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:13:04.367204 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:13:04.367413 (+ 209us) client_negotiation.cc:769] Sending connection context
0929 01:13:04.367664 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 01:13:04.368220 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":133,"thread_start_us":395,"threads_started":1}
I20260929 01:13:04.372721 28766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.253@KRBTEST.COM" created.
I20260929 01:13:04.373911 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.373512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:43107 (local address 127.0.0.1:35776)
0929 01:13:04.373689 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.373801 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:43107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 01:13:04.374300 28715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.198.253:43107: connect: Connection refused (error 111)
I20260929 01:13:04.375921 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.375540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:43107 (local address 127.0.0.1:35788)
0929 01:13:04.375676 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.375742 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:43107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 01:13:04.376161 28766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.253:43107: Network error: Client connection negotiation failed: client connection to 127.5.198.253:43107: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.253.keytab.
Entry for principal kudu/127.5.198.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.253.keytab.
I20260929 01:13:04.403795 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.253:43107
--webserver_interface=127.5.198.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.253.keytab
--principal=kudu/127.5.198.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:34893,127.5.198.253:43107,127.5.198.252:38157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:13:04.425498 28766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } attempt: 1
I20260929 01:13:04.426796 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.426326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:43107 (local address 127.0.0.1:35800)
0929 01:13:04.426549 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.426662 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:43107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 01:13:04.427784 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.427458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:43107 (local address 127.0.0.1:35808)
0929 01:13:04.427601 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.427670 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:43107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 01:13:04.427964 28766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.253:43107: Network error: Client connection negotiation failed: client connection to 127.5.198.253:43107: connect: Connection refused (error 111)
I20260929 01:13:04.496237 28766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } attempt: 2
I20260929 01:13:04.497488 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.497000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:43107 (local address 127.0.0.1:35818)
0929 01:13:04.497259 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.497346 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:43107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 01:13:04.498471 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.498136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:43107 (local address 127.0.0.1:35834)
0929 01:13:04.498270 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.498349 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:43107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 01:13:04.498668 28766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.253:43107: Network error: Client connection negotiation failed: client connection to 127.5.198.253:43107: connect: Connection refused (error 111)
I20260929 01:13:04.589915 28766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } attempt: 3
I20260929 01:13:04.591084 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.590647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:43107 (local address 127.0.0.1:35840)
0929 01:13:04.590849 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.590951 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:43107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 01:13:04.592072 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.591734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:43107 (local address 127.0.0.1:35842)
0929 01:13:04.591872 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.591947 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:43107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 01:13:04.592309 28766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.253:43107: Network error: Client connection negotiation failed: client connection to 127.5.198.253:43107: connect: Connection refused (error 111)
I20260929 01:13:04.735569 28766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } attempt: 4
W20260929 01:13:04.736397 28773 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:13:04.736896 28773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:13:04.737002 28773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260929 01:13:04.737126 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.736271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:43107 (local address 127.0.0.1:35846)
0929 01:13:04.736780 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.736879 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:43107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305}
W20260929 01:13:04.737150 28773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 01:13:04.738124 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.737771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:43107 (local address 127.0.0.1:35862)
0929 01:13:04.737913 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.738010 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.253:43107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 01:13:04.738328 28766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.253:43107: Network error: Client connection negotiation failed: client connection to 127.5.198.253:43107: connect: Connection refused (error 111)
W20260929 01:13:04.744946 28773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:13:04.745025 28773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:13:04.745160 28773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:13:04.745247 28773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:13:04.745337 28773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:13:04.755255 28773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:34893,127.5.198.253:43107,127.5.198.252:38157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.253.keytab
--principal=kudu/127.5.198.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.253:43107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.198.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:13:04.756625 28773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:13:04.757983 28773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:13:04.769939 28778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:13:04.770099 28779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:13:04.770368 28781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:13:04.770982 28773 server_base.cc:1034] running on GCE node
I20260929 01:13:04.771703 28773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:13:04.773170 28773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:13:04.774477 28773 hybrid_clock.cc:648] HybridClock initialized: now 1790644384774347 us; error 60 us; skew 500 ppm
I20260929 01:13:04.774806 28773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:13:04 dist-test-slave-41cl krb5kdc[28686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644384, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:13:04.778168 28773 init.cc:377] Logged in from keytab as kudu/127.5.198.253@KRBTEST.COM (short username kudu)
I20260929 01:13:04.779680 28773 webserver.cc:460] Webserver started at http://127.5.198.253:40203/ using document root <none> and password file <none>
I20260929 01:13:04.780344 28773 fs_manager.cc:359] Metadata directory not provided
I20260929 01:13:04.780521 28773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:13:04.780845 28773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:13:04.783774 28773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/data/instance:
uuid: "60097088018443b6bc7a95c830f5a87b"
format_stamp: "Formatted at 2026-09-29 01:13:04 on dist-test-slave-41cl"
server_key: "c328085f458d8a604c2eef9a61e48a74"
server_key_iv: "e0184380733c4c5825d239a821047a41"
server_key_version: "encryptionkey@0"
I20260929 01:13:04.784621 28773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/wal/instance:
uuid: "60097088018443b6bc7a95c830f5a87b"
format_stamp: "Formatted at 2026-09-29 01:13:04 on dist-test-slave-41cl"
server_key: "c328085f458d8a604c2eef9a61e48a74"
server_key_iv: "e0184380733c4c5825d239a821047a41"
server_key_version: "encryptionkey@0"
I20260929 01:13:04.789932 28773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 01:13:04.793238 28788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:13:04.794538 28773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:13:04.794731 28773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/wal
uuid: "60097088018443b6bc7a95c830f5a87b"
format_stamp: "Formatted at 2026-09-29 01:13:04 on dist-test-slave-41cl"
server_key: "c328085f458d8a604c2eef9a61e48a74"
server_key_iv: "e0184380733c4c5825d239a821047a41"
server_key_version: "encryptionkey@0"
I20260929 01:13:04.794945 28773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:13:04.811880 28773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:13:04.815716 28773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:13:04.815987 28773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:13:04.834704 28773 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.253:43107
I20260929 01:13:04.834726 28839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.253:43107 every 8 connection(s)
I20260929 01:13:04.836020 28773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/data/info.pb
I20260929 01:13:04.839205 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 28773
I20260929 01:13:04.839386 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-1/wal/instance
I20260929 01:13:04.839751 28840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:13:04.839875 5915 external_mini_cluster.cc:1428] Setting key e90222756fa7a04a6604c5b04bcea05e
I20260929 01:13:04.846175 28840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:13:04 dist-test-slave-41cl krb5kdc[28686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644384, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.253@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.198.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.198.252@KRBTEST.COM" created.
I20260929 01:13:04.871830 28769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.848730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33216 (local address 127.5.198.254:34893)
0929 01:13:04.848945 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:04.848951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:04.850206 (+ 1255us) server_negotiation.cc:407] Connection header received
0929 01:13:04.852041 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:04.852047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:04.852137 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:04.852243 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:04.853983 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:04.855063 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:04.857250 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:04.857871 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:04.857966 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:04.865172 (+ 7206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:04.865177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:04.865181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:04.865217 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:04.868388 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:04.869219 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:04.869227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:04.869233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:04.869386 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:04.869808 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:04.869814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:04.869817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:04.870215 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:04.870379 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:04.871149 (+ 770us) server_negotiation.cc:299] Negotiation successful
0929 01:13:04.871401 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:13:04.872639 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.849261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:34893 (local address 127.0.0.1:33216)
0929 01:13:04.850025 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.850070 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 01:13:04.851778 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:13:04.852527 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:13:04.852536 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:13:04.853028 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:13:04.853787 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:04.853810 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:04.855270 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:04.855275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:04.857083 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:04.857098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:04.858281 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:04.858287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:04.858590 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:04.861531 (+ 2941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:13:04.861562 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:13:04.864850 (+ 3288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:13:04.868604 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:04.868612 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:04.868627 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:04.869069 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:04.869506 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:04.869509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:04.869511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:04.869684 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:04.870407 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:13:04.870413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:13:04.870750 (+ 337us) client_negotiation.cc:769] Sending connection context
0929 01:13:04.870999 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 01:13:04.871384 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":336,"threads_started":1}
I20260929 01:13:04.874522 28840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:13:04 dist-test-slave-41cl krb5kdc[28686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644384, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.253@KRBTEST.COM for kudu/127.5.198.253@KRBTEST.COM
I20260929 01:13:04.890878 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.875324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:43107 (local address 127.0.0.1:35878)
0929 01:13:04.875558 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.875577 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:13:04.875677 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:13:04.876755 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:13:04.876760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:13:04.877266 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:13:04.877439 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:04.877455 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:04.878937 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:04.878946 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:04.879873 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:04.879884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:04.880817 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:04.880822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:04.881026 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:04.881683 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:13:04.881702 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:13:04.884114 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:13:04.887837 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:04.887845 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:04.887850 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:04.888407 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:04.888972 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:04.888976 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:04.888979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:04.889182 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:04.890152 (+ 970us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:13:04.890156 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:13:04.890272 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 01:13:04.890388 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 01:13:04.890605 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":3}
I20260929 01:13:04.891453 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.875607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.5.198.253:43107)
0929 01:13:04.876251 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:04.876257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:04.876283 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:13:04.876331 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:04.876337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:04.876604 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:04.876773 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:04.877635 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:04.878793 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:04.880021 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:04.880665 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:04.880813 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:04.884362 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:04.884375 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:04.884389 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:04.884420 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:04.887685 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:04.888602 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:04.888615 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:04.888622 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:04.888798 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:04.889316 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:04.889325 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:04.889329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:04.889973 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:04.890211 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:04.890642 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 01:13:04.890969 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":373,"threads_started":1}
I20260929 01:13:04.893909 28840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:13:04.895431 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.894887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44142)
0929 01:13:04.895088 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.895256 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260929 01:13:04.895723 28790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
I20260929 01:13:04.897397 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.896994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44156)
0929 01:13:04.897216 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.897297 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 01:13:04.897634 28840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:38157: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.198.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.252.keytab.
Entry for principal kudu/127.5.198.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.252.keytab.
I20260929 01:13:04.903486 5915 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
/tmp/dist-test-taskd3rvDY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.252:38157
--webserver_interface=127.5.198.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.252.keytab
--principal=kudu/127.5.198.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.198.212:34581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.198.254:34893,127.5.198.253:43107,127.5.198.252:38157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:13:04.946992 28840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } attempt: 1
I20260929 01:13:04.948335 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.947774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44162)
0929 01:13:04.948057 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.948163 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 01:13:04.949451 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:04.949078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44170)
0929 01:13:04.949261 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:04.949336 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 01:13:04.949755 28840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:38157: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
I20260929 01:13:05.017999 28840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } attempt: 2
I20260929 01:13:05.019215 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.018749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44180)
0929 01:13:05.018968 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.019057 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 01:13:05.020262 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.019942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44188)
0929 01:13:05.020077 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.020146 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 01:13:05.020620 28840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:38157: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
I20260929 01:13:05.037611 28766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } attempt: 5
Sep 29 01:13:05 dist-test-slave-41cl krb5kdc[28686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644384, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for kudu/127.5.198.253@KRBTEST.COM
I20260929 01:13:05.051893 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.038501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:43107 (local address 127.0.0.1:35894)
0929 01:13:05.038706 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.038730 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:13:05.038900 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:13:05.039311 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:13:05.039317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:13:05.039733 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:13:05.039949 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:05.039961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.041131 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.041140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:05.042036 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:05.042050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.042850 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.042856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:05.043097 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.043805 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:13:05.043832 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:13:05.046227 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:13:05.049523 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:05.049529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:05.049533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:05.049901 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:05.050274 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:05.050279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:05.050283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:05.050429 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:05.050941 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:13:05.050946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:13:05.051105 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 01:13:05.051256 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 01:13:05.051546 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 01:13:05.051898 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.038725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.5.198.253:43107)
0929 01:13:05.038935 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.038944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.038970 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:13:05.039041 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.039049 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.039141 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.039258 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.040118 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.040979 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.042210 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.042678 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.042839 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.046440 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:05.046445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:05.046448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:05.046479 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:05.049379 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.050037 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.050047 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.050049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.050162 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.050539 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.050545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.050548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.050821 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:05.050985 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:05.051351 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 01:13:05.051558 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 01:13:05.053326 28766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:13:05.054350 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.053992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44196)
0929 01:13:05.054145 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.054218 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 01:13:05.055192 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.054892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44202)
0929 01:13:05.055027 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.055107 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 01:13:05.055374 28766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:38157: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
I20260929 01:13:05.106616 28766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } attempt: 1
I20260929 01:13:05.107769 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.107299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44206)
0929 01:13:05.107525 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.107609 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 01:13:05.108716 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.108372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44218)
0929 01:13:05.108540 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.108619 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 01:13:05.108928 28766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:38157: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
I20260929 01:13:05.111896 28840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } attempt: 3
I20260929 01:13:05.113030 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.112572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44224)
0929 01:13:05.112746 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.112838 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 01:13:05.114071 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.113667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44228)
0929 01:13:05.113895 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.113966 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260929 01:13:05.114295 28840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:38157: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
I20260929 01:13:05.177192 28766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } attempt: 2
I20260929 01:13:05.178578 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.178078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44240)
0929 01:13:05.178327 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.178422 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 01:13:05.179529 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.179214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44256)
0929 01:13:05.179361 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.179430 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 01:13:05.179785 28766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:38157: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
W20260929 01:13:05.253464 28847 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:13:05.253801 28847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:13:05.253916 28847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:13:05.254057 28847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 01:13:05.257578 28840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } attempt: 4
I20260929 01:13:05.258836 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.258333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44260)
0929 01:13:05.258589 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.258684 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 01:13:05.259876 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.259556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44270)
0929 01:13:05.259697 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.259774 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 01:13:05.260106 28840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:38157: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
W20260929 01:13:05.262331 28847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:13:05.262444 28847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:13:05.262560 28847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:13:05.262645 28847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:13:05.262751 28847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:13:05.272668 28847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.198.212:34581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.198.254:34893,127.5.198.253:43107,127.5.198.252:38157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/krb5kdc/kudu_127.5.198.252.keytab
--principal=kudu/127.5.198.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.198.252:38157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.198.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:13:05.274030 28847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:13:05.275326 28847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:13:05.285969 28766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } attempt: 3
I20260929 01:13:05.287335 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.286612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44284)
0929 01:13:05.286972 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.287074 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260929 01:13:05.287995 28853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:13:05.288326 28852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:13:05.288359 28855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:13:05.288767 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.288238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44286)
0929 01:13:05.288553 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.288642 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260929 01:13:05.288985 28847 server_base.cc:1034] running on GCE node
I20260929 01:13:05.289750 28847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260929 01:13:05.291180 28766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.198.252:38157: Network error: Client connection negotiation failed: client connection to 127.5.198.252:38157: connect: Connection refused (error 111)
I20260929 01:13:05.291220 28847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:13:05.292459 28847 hybrid_clock.cc:648] HybridClock initialized: now 1790644385292390 us; error 36 us; skew 500 ppm
I20260929 01:13:05.292850 28847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:13:05 dist-test-slave-41cl krb5kdc[28686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644385, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:13:05.296027 28847 init.cc:377] Logged in from keytab as kudu/127.5.198.252@KRBTEST.COM (short username kudu)
I20260929 01:13:05.297521 28847 webserver.cc:460] Webserver started at http://127.5.198.252:35499/ using document root <none> and password file <none>
I20260929 01:13:05.298218 28847 fs_manager.cc:359] Metadata directory not provided
I20260929 01:13:05.298350 28847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:13:05.298671 28847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:13:05.301623 28847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/data/instance:
uuid: "3d928c2fbfd64e0c8b71f6ede294eb27"
format_stamp: "Formatted at 2026-09-29 01:13:05 on dist-test-slave-41cl"
server_key: "bf63fb7664fbca0a4cbfda950008e995"
server_key_iv: "d33d7633621e923cf318f7c761472b74"
server_key_version: "encryptionkey@0"
I20260929 01:13:05.302387 28847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/wal/instance:
uuid: "3d928c2fbfd64e0c8b71f6ede294eb27"
format_stamp: "Formatted at 2026-09-29 01:13:05 on dist-test-slave-41cl"
server_key: "bf63fb7664fbca0a4cbfda950008e995"
server_key_iv: "d33d7633621e923cf318f7c761472b74"
server_key_version: "encryptionkey@0"
I20260929 01:13:05.307564 28847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260929 01:13:05.311180 28862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:13:05.312662 28847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:13:05.312852 28847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/data,/tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/wal
uuid: "3d928c2fbfd64e0c8b71f6ede294eb27"
format_stamp: "Formatted at 2026-09-29 01:13:05 on dist-test-slave-41cl"
server_key: "bf63fb7664fbca0a4cbfda950008e995"
server_key_iv: "d33d7633621e923cf318f7c761472b74"
server_key_version: "encryptionkey@0"
I20260929 01:13:05.313076 28847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:13:05.336815 28847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:13:05.341298 28847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:13:05.341596 28847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:13:05.360761 28847 rpc_server.cc:307] RPC server started. Bound to: 127.5.198.252:38157
I20260929 01:13:05.360795 28913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.198.252:38157 every 8 connection(s)
I20260929 01:13:05.362152 28847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/data/info.pb
I20260929 01:13:05.365887 28914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:13:05.368744 5915 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu as pid 28847
I20260929 01:13:05.368929 5915 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskd3rvDY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790644023292677-5915-0/minicluster-data/master-2/wal/instance
I20260929 01:13:05.369486 5915 external_mini_cluster.cc:1428] Setting key 9549d15c4ed1e0206695f0bf2a22c3bf
I20260929 01:13:05.373926 28914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:13:05 dist-test-slave-41cl krb5kdc[28686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.252@KRBTEST.COM
Sep 29 01:13:05 dist-test-slave-41cl krb5kdc[28686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644385, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.252@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:13:05.392023 28918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.376784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33230 (local address 127.5.198.254:34893)
0929 01:13:05.377307 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.377315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.377794 (+ 479us) server_negotiation.cc:407] Connection header received
0929 01:13:05.378004 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.378010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.378083 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.378198 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.379231 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.379965 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.381618 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.382032 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.382166 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.385404 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:05.385412 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:05.385419 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:05.385463 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:05.388795 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.389681 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.389686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.389689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.389830 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.390268 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.390272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.390275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.390648 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:05.390808 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:05.391367 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 01:13:05.391597 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":280,"threads_started":1}
I20260929 01:13:05.392446 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.376899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.254:34893 (local address 127.0.0.1:33230)
0929 01:13:05.377658 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.377697 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:13:05.377876 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:13:05.378263 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:13:05.378269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:13:05.378847 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:13:05.379088 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:05.379096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.380124 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.380136 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:05.381466 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:05.381478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.382252 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.382259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:05.382410 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.383002 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:13:05.383024 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:13:05.385216 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:13:05.388968 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:05.388977 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:05.388982 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:05.389570 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:05.389976 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:05.389979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:05.389981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:05.390132 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:05.390809 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:13:05.390829 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:13:05.391041 (+ 212us) client_negotiation.cc:769] Sending connection context
0929 01:13:05.391259 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 01:13:05.391574 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":397,"threads_started":1}
I20260929 01:13:05.392518 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.371323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.5.198.252:38157)
0929 01:13:05.372122 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.372151 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.372192 (+ 41us) server_negotiation.cc:407] Connection header received
0929 01:13:05.373806 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.373828 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.374238 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.374578 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.376294 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.378249 (+ 1955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.379515 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.380118 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.380351 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.384233 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:05.384241 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:05.384251 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:05.384284 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:05.388051 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.388787 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.388796 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.388804 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.388996 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.389426 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.389429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.389431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.390124 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:05.390424 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:05.390807 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 01:13:05.391358 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":365,"threads_started":1}
I20260929 01:13:05.394068 28914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:13:05 dist-test-slave-41cl krb5kdc[28686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644385, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.252@KRBTEST.COM for kudu/127.5.198.253@KRBTEST.COM
Sep 29 01:13:05 dist-test-slave-41cl krb5kdc[28686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.254@KRBTEST.COM
I20260929 01:13:05.408174 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.394805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.253:43107 (local address 127.0.0.1:35900)
0929 01:13:05.395099 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.395113 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:13:05.395235 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:13:05.395728 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:13:05.395731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:13:05.396075 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:13:05.396261 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:05.396275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.397350 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.397354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:05.398348 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:05.398362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.399321 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.399328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:05.399609 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.400263 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:13:05.400280 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:13:05.402601 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:13:05.405781 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:05.405786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:05.405790 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:05.406155 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:05.406556 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:05.406561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:05.406565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:05.406708 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:05.407217 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:13:05.407222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:13:05.407383 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 01:13:05.407524 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 01:13:05.407798 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260929 01:13:05.408198 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.394827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35900 (local address 127.5.198.253:43107)
0929 01:13:05.395021 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.395027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.395185 (+ 158us) server_negotiation.cc:407] Connection header received
0929 01:13:05.395362 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.395373 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.395473 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.395581 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.396435 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.397183 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.398641 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.399146 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.399242 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.402792 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:05.402796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:05.402800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:05.402832 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:05.405645 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.406282 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.406288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.406292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.406441 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.406818 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.406824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.406828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.407090 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:05.407253 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:05.407615 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 01:13:05.407818 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 01:13:05.409268 28914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:13:05.410985 28918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.395498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33232 (local address 127.5.198.254:34893)
0929 01:13:05.395847 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.395851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.395863 (+ 12us) server_negotiation.cc:407] Connection header received
0929 01:13:05.395899 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.395902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.395966 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.396035 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.397314 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.398044 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.399709 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.400343 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.400456 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.404852 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:05.404857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:05.404860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:05.404892 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:05.407771 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.408773 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.408780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.408784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.408913 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.409332 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.409336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.409339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.409704 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:05.409989 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:05.410429 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 01:13:05.410651 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
Sep 29 01:13:05 dist-test-slave-41cl krb5kdc[28686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644385, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.252@KRBTEST.COM for kudu/127.5.198.252@KRBTEST.COM
Sep 29 01:13:05 dist-test-slave-41cl krb5kdc[28686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.198.253@KRBTEST.COM
I20260929 01:13:05.423235 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.410010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44302)
0929 01:13:05.410162 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.410177 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:13:05.410280 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:13:05.410694 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:13:05.410698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:13:05.411182 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:13:05.411321 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:05.411329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.412440 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.412445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:05.413395 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:05.413407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.414080 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.414085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:05.414297 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.414877 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:13:05.414893 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:13:05.416868 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:13:05.420822 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:05.420827 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:05.420831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:05.421223 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:05.421568 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:05.421575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:05.421579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:05.421703 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:05.422315 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:13:05.422319 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:13:05.422446 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 01:13:05.422561 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 01:13:05.422899 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 01:13:05.423488 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.410241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44302 (local address 127.5.198.252:38157)
0929 01:13:05.410392 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.410397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.410413 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:13:05.410467 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.410472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.410550 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.410649 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.411456 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.412313 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.413542 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.413933 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.414076 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.417044 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:05.417049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:05.417061 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:05.417111 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:05.420695 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.421337 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.421345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.421348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.421465 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.421804 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.421812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.421815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.422187 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:05.422333 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:05.422690 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 01:13:05.423098 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":54}
I20260929 01:13:05.427990 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.412743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35916 (local address 127.5.198.253:43107)
0929 01:13:05.412898 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.412904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.413036 (+ 132us) server_negotiation.cc:407] Connection header received
0929 01:13:05.413217 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.413222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.413289 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.413391 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.414536 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.415237 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.416540 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.417029 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.417123 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.422063 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:05.422069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:05.422074 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:05.422112 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:05.425252 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.425875 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.425882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.425885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.426040 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.426479 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.426484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.426488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.426856 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:05.427157 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:05.427427 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 01:13:05.427663 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:13:05.429786 28914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27: Bootstrap starting.
I20260929 01:13:05.433106 5915 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:13:05.433602 28914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27: Neither blocks nor log segments found. Creating new log.
I20260929 01:13:05.434865 28914 log.cc:824] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27: Log is configured to *not* fsync() on all Append() calls
I20260929 01:13:05.437783 28914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27: No bootstrap required, opened a new log
I20260929 01:13:05.436170 28741 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9edb94110ac644a8b638f314696a91da"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33232
I20260929 01:13:05.444844 28914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } }
I20260929 01:13:05.445190 28914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d928c2fbfd64e0c8b71f6ede294eb27, State: Initialized, Role: FOLLOWER
I20260929 01:13:05.445781 28914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } }
I20260929 01:13:05.446784 28921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } } }
I20260929 01:13:05.447180 28921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:13:05.447917 28914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:13:05.457281 28932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:13:05.457455 28932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:13:05.455430 28847 master_runner.cc:186] Error getting master registration for 127.5.198.254:34893: OK, instance_id { permanent_uuid: "9edb94110ac644a8b638f314696a91da" instance_seqno: 1790644384328777 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:13:05.464026 28847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:13:05.468411 28766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } attempt: 4
Sep 29 01:13:05 dist-test-slave-41cl krb5kdc[28686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644384, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.254@KRBTEST.COM for kudu/127.5.198.252@KRBTEST.COM
I20260929 01:13:05.485565 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.469137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44306)
0929 01:13:05.469399 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.469433 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 01:13:05.469585 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:13:05.469992 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:13:05.469997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:13:05.470570 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:13:05.470793 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:05.470807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.472298 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.472305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:05.473827 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:05.473844 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.474943 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.474950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:05.475170 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.475891 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:13:05.475926 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:13:05.478813 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:13:05.482956 (+ 4143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:05.482962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:05.482966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:05.483335 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:05.483710 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:05.483715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:05.483719 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:05.483863 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:05.484446 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:13:05.484453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:13:05.484644 (+ 191us) client_negotiation.cc:769] Sending connection context
0929 01:13:05.484826 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 01:13:05.485148 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 01:13:05.485590 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.469180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44306 (local address 127.5.198.252:38157)
0929 01:13:05.469398 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.469414 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.469539 (+ 125us) server_negotiation.cc:407] Connection header received
0929 01:13:05.469723 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.469731 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.469818 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.469923 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.470961 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.472127 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.474017 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.474742 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.474903 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.479093 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:05.479101 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:05.479106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:05.479146 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:05.482803 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.483467 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.483473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.483476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.483599 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.483971 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.483977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.483981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.484302 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:05.484518 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:05.484932 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 01:13:05.485147 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:13:05.491667 28766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da: Bootstrap starting.
I20260929 01:13:05.495760 28766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da: Neither blocks nor log segments found. Creating new log.
I20260929 01:13:05.497144 28766 log.cc:824] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da: Log is configured to *not* fsync() on all Append() calls
I20260929 01:13:05.500555 28766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da: No bootstrap required, opened a new log
I20260929 01:13:05.508181 28766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } }
I20260929 01:13:05.508616 28766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9edb94110ac644a8b638f314696a91da, State: Initialized, Role: FOLLOWER
I20260929 01:13:05.509145 28766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } }
I20260929 01:13:05.510106 28935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } } }
I20260929 01:13:05.510499 28935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:13:05.511332 28766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:13:05.511375 28741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 01:13:05.511600 28741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:13:05.514730 28741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } }
I20260929 01:13:05.517027 28741 leader_election.cc:288] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [CANDIDATE]: Term 1 election: Requested vote from peers 60097088018443b6bc7a95c830f5a87b (127.5.198.253:43107), 3d928c2fbfd64e0c8b71f6ede294eb27 (127.5.198.252:38157)
I20260929 01:13:05.517547 28889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9edb94110ac644a8b638f314696a91da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27"
I20260929 01:13:05.518055 28889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:13:05.521461 28889 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edb94110ac644a8b638f314696a91da in term 1.
I20260929 01:13:05.522107 28716 leader_election.cc:302] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d928c2fbfd64e0c8b71f6ede294eb27, 9edb94110ac644a8b638f314696a91da; no voters:
I20260929 01:13:05.522703 28935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:13:05.524886 28935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [term 1 LEADER]: Becoming Leader. State: Replica: 9edb94110ac644a8b638f314696a91da, State: Running, Role: LEADER
I20260929 01:13:05.525579 28935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } }
I20260929 01:13:05.528537 28946 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:13:05.532279 28948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9edb94110ac644a8b638f314696a91da. Latest consensus state: current_term: 1 leader_uuid: "9edb94110ac644a8b638f314696a91da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } } }
I20260929 01:13:05.532787 28948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [sys.catalog]: This master's current role is: LEADER
I20260929 01:13:05.545051 28952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:13:05.534792 28699 master_runner.cc:186] Error getting master registration for 127.5.198.253:43107: OK, instance_id { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" instance_seqno: 1790644384829328 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:13:05.546753 28699 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:13:05.552443 28952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:13:05.559296 28840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } attempt: 5
W20260929 01:13:05.568037 28720 tablet.cc:2369] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:13:05.573061 28918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.549424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33246 (local address 127.5.198.254:34893)
0929 01:13:05.549646 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.549653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.550345 (+ 692us) server_negotiation.cc:407] Connection header received
0929 01:13:05.550695 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.550700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.550784 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.550880 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.553330 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.554444 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.556341 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.557303 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.557426 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.563035 (+ 5609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:05.563045 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:05.563053 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:05.563127 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:05.568997 (+ 5870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.570052 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.570061 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.570067 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.570255 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.570761 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.570769 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.570775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.571208 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:05.571540 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:05.572192 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 01:13:05.572497 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 01:13:05.582448 28889 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27 [term 1 FOLLOWER]: Refusing update from remote peer 9edb94110ac644a8b638f314696a91da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:13:05.583357 28948 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 29 01:13:05 dist-test-slave-41cl krb5kdc[28686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644384, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.198.253@KRBTEST.COM for kudu/127.5.198.252@KRBTEST.COM
W20260929 01:13:05.593698 28868 tablet.cc:2369] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:13:05.596145 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.550542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35918 (local address 127.5.198.253:43107)
0929 01:13:05.550717 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.550723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.557020 (+ 6297us) server_negotiation.cc:407] Connection header received
0929 01:13:05.557301 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.557310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.557414 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.557495 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.563122 (+ 5627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.564239 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.571320 (+ 7081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.571995 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.572103 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.586999 (+ 14896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:05.587007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:05.587013 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:05.587056 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:05.592524 (+ 5468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.593372 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.593378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.593381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.593554 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.594114 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.594122 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.594128 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.594499 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:05.594741 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:05.595354 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 01:13:05.595757 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:13:05.596443 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.551067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44314 (local address 127.5.198.252:38157)
0929 01:13:05.551251 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.551258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.560620 (+ 9362us) server_negotiation.cc:407] Connection header received
0929 01:13:05.560867 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.560875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.560972 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.561047 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.568302 (+ 7255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.569389 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.583040 (+ 13651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.583723 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.583821 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.589576 (+ 5755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:05.589583 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:05.589587 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:05.589627 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:05.593208 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.594013 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.594019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.594022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.594180 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.594650 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.594655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.594658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.595015 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:05.595240 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:05.595805 (+ 565us) server_negotiation.cc:299] Negotiation successful
0929 01:13:05.596055 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 01:13:05.610232 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.559890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.198.252:38157 (local address 127.0.0.1:44320)
0929 01:13:05.560338 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 01:13:05.560360 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:13:05.560496 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:13:05.575679 (+ 15183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:13:05.575686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:13:05.576192 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:13:05.576398 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:05.576409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.580625 (+ 4216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.580631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:05.581946 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:13:05.581962 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.588694 (+ 6732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.588701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:13:05.588958 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.589492 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:13:05.589515 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:13:05.592330 (+ 2815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:13:05.600675 (+ 8345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:05.600681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:05.600686 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:05.601233 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:05.601751 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:13:05.601760 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:13:05.601766 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:13:05.601983 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:13:05.609169 (+ 7186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:13:05.609175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:13:05.609348 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 01:13:05.609499 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 01:13:05.609836 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331}
I20260929 01:13:05.611344 28959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:05.559952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44320 (local address 127.5.198.252:38157)
0929 01:13:05.575345 (+ 15393us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.575351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.575374 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:13:05.575445 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.575450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.575535 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.575644 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.579149 (+ 3505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.580492 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.584070 (+ 3578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:05.588450 (+ 4380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:05.594979 (+ 6529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:05.595397 (+ 418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:13:05.595412 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:13:05.595417 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:13:05.595476 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:13:05.600519 (+ 5043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.601397 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.601403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.601406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.601577 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:13:05.608558 (+ 6981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:13:05.608564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:13:05.608568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:13:05.609003 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:13:05.610518 (+ 1515us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:05.610674 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 01:13:05.610922 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15260,"thread_start_us":15029,"threads_started":1}
I20260929 01:13:05.615159 28921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9edb94110ac644a8b638f314696a91da. Latest consensus state: current_term: 1 leader_uuid: "9edb94110ac644a8b638f314696a91da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } } }
I20260929 01:13:05.615482 28921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:13:05.617950 28840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b: Bootstrap starting.
I20260929 01:13:05.625268 28840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b: Neither blocks nor log segments found. Creating new log.
I20260929 01:13:05.627516 28840 log.cc:824] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:13:05.631945 28958 mvcc.cc:205] Tried to move back new op lower bound from 7334479403328962560 to 7334479403120209920. Current Snapshot: MvccSnapshot[applied={T|T < 7334479403328962560}]
I20260929 01:13:05.633184 28960 mvcc.cc:205] Tried to move back new op lower bound from 7334479403328962560 to 7334479403120209920. Current Snapshot: MvccSnapshot[applied={T|T < 7334479403328962560}]
I20260929 01:13:05.637719 28840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b: No bootstrap required, opened a new log
I20260929 01:13:05.646174 28921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9edb94110ac644a8b638f314696a91da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } } }
I20260929 01:13:05.646476 28921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:13:05.650938 28840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } }
I20260929 01:13:05.651453 28840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60097088018443b6bc7a95c830f5a87b, State: Initialized, Role: FOLLOWER
I20260929 01:13:05.651995 28840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } }
I20260929 01:13:05.653004 28948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9edb94110ac644a8b638f314696a91da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } } }
I20260929 01:13:05.653309 28948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [sys.catalog]: This master's current role is: LEADER
I20260929 01:13:05.658927 28965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } } }
I20260929 01:13:05.659334 28965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:13:05.661535 28840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:13:05.661580 28815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9edb94110ac644a8b638f314696a91da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "60097088018443b6bc7a95c830f5a87b"
I20260929 01:13:05.661927 28814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:13:05.666679 28814 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b [term 1 FOLLOWER]: Refusing update from remote peer 9edb94110ac644a8b638f314696a91da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:13:05.670818 28948 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [LEADER]: Connected to new peer: Peer: permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:13:05.672165 28952 catalog_manager.cc:1347] Generated new cluster ID: 34f13e98b40d4e378641c535030c003f
I20260929 01:13:05.672255 28952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:13:05.673381 28815 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edb94110ac644a8b638f314696a91da in term 1.
I20260929 01:13:05.722579 28965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9edb94110ac644a8b638f314696a91da. Latest consensus state: current_term: 1 leader_uuid: "9edb94110ac644a8b638f314696a91da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } } }
I20260929 01:13:05.722951 28965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:13:05.723237 28974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9edb94110ac644a8b638f314696a91da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } } }
I20260929 01:13:05.723596 28974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:13:05.755769 28980 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:13:05.755765 28948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9edb94110ac644a8b638f314696a91da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edb94110ac644a8b638f314696a91da" member_type: VOTER last_known_addr { host: "127.5.198.254" port: 34893 } } peers { permanent_uuid: "60097088018443b6bc7a95c830f5a87b" member_type: VOTER last_known_addr { host: "127.5.198.253" port: 43107 } } peers { permanent_uuid: "3d928c2fbfd64e0c8b71f6ede294eb27" member_type: VOTER last_known_addr { host: "127.5.198.252" port: 38157 } } }
I20260929 01:13:05.756039 28948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da [sys.catalog]: This master's current role is: LEADER
I20260929 01:13:05.768185 28952 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:13:05.770329 28952 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:13:05.779384 28952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9edb94110ac644a8b638f314696a91da: Generated new TSK 0
I20260929 01:13:05.782161 28952 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 01:13:05.891860 28918 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:05.878367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.5.198.254:34893)
0929 01:13:05.878609 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.878618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.889940 (+ 11322us) server_negotiation.cc:407] Connection header received
0929 01:13:05.890202 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.890207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.890294 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.890363 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.891580 (+ 1217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33248: BlockingRecv error: recv got EOF from 127.0.0.1:33248 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 01:13:05.894004 28959 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:05.889683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.5.198.252:38157)
0929 01:13:05.889890 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.889896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.892450 (+ 2554us) server_negotiation.cc:407] Connection header received
0929 01:13:05.892716 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.892722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.892821 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.892890 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.893768 (+ 878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44328: BlockingRecv error: recv got EOF from 127.0.0.1:44328 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 01:13:05.895541 28844 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:05.878543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.5.198.253:43107)
0929 01:13:05.878730 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.878736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.888966 (+ 10230us) server_negotiation.cc:407] Connection header received
0929 01:13:05.889221 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.889227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.889332 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.889438 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.895313 (+ 5875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35922: BlockingRecv error: recv got EOF from 127.0.0.1:35922 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 01:13:05.904238 28844 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:05.901817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35928 (local address 127.5.198.253:43107)
0929 01:13:05.901999 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.902005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.902497 (+ 492us) server_negotiation.cc:407] Connection header received
0929 01:13:05.902748 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.902754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.902877 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.902952 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.903949 (+ 997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35928: BlockingRecv error: recv got EOF from 127.0.0.1:35928 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 01:13:05.910413 28844 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:05.908363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.5.198.253:43107)
0929 01:13:05.908552 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.908558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.909052 (+ 494us) server_negotiation.cc:407] Connection header received
0929 01:13:05.909227 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.909233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.909324 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.909442 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.910204 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35934: BlockingRecv error: recv got EOF from 127.0.0.1:35934 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 01:13:05.920393 28844 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:05.918247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35942 (local address 127.5.198.253:43107)
0929 01:13:05.918406 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.918412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.918908 (+ 496us) server_negotiation.cc:407] Connection header received
0929 01:13:05.919095 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.919102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.919211 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.919327 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.920148 (+ 821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35942: BlockingRecv error: recv got EOF from 127.0.0.1:35942 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 01:13:05.932269 28844 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:05.930093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35956 (local address 127.5.198.253:43107)
0929 01:13:05.930323 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.930329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.930813 (+ 484us) server_negotiation.cc:407] Connection header received
0929 01:13:05.930982 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.930995 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.931088 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.931202 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.932034 (+ 832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35956: BlockingRecv error: recv got EOF from 127.0.0.1:35956 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260929 01:13:05.945585 28844 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:05.943452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.5.198.253:43107)
0929 01:13:05.943632 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.943638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.944092 (+ 454us) server_negotiation.cc:407] Connection header received
0929 01:13:05.944291 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.944298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.944384 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.944550 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.945349 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35972: BlockingRecv error: recv got EOF from 127.0.0.1:35972 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 01:13:05.972486 28844 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:05.970206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35982 (local address 127.5.198.253:43107)
0929 01:13:05.970400 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:05.970407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:05.970955 (+ 548us) server_negotiation.cc:407] Connection header received
0929 01:13:05.971153 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:05.971159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:05.971267 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:05.971380 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:05.972221 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35982: BlockingRecv error: recv got EOF from 127.0.0.1:35982 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 01:13:06.012663 28844 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:06.010288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35990 (local address 127.5.198.253:43107)
0929 01:13:06.010519 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:06.010525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:06.011064 (+ 539us) server_negotiation.cc:407] Connection header received
0929 01:13:06.011300 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:06.011306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:06.011400 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:06.011499 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:06.012375 (+ 876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35990: BlockingRecv error: recv got EOF from 127.0.0.1:35990 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 01:13:06.082580 28844 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:06.080626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.5.198.253:43107)
0929 01:13:06.080845 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:06.080852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:06.081123 (+ 271us) server_negotiation.cc:407] Connection header received
0929 01:13:06.081345 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:06.081350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:06.081426 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:06.081520 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:06.082371 (+ 851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35994: BlockingRecv error: recv got EOF from 127.0.0.1:35994 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260929 01:13:06.217356 28844 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:06.215280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.5.198.253:43107)
0929 01:13:06.215560 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:06.215568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:06.215829 (+ 261us) server_negotiation.cc:407] Connection header received
0929 01:13:06.215988 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:06.215992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:06.216071 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:06.216217 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:06.217092 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36002: BlockingRecv error: recv got EOF from 127.0.0.1:36002 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 01:13:06.459375 28932 catalog_manager.cc:1259] Loaded cluster ID: 34f13e98b40d4e378641c535030c003f
I20260929 01:13:06.459474 28932 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27: loading cluster ID for follower catalog manager: success
I20260929 01:13:06.462064 28932 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27: acquiring CA information for follower catalog manager: success
I20260929 01:13:06.463860 28932 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3d928c2fbfd64e0c8b71f6ede294eb27: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:13:06.478492 28844 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:13:06.476753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36004 (local address 127.5.198.253:43107)
0929 01:13:06.476990 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:06.476996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:06.477156 (+ 160us) server_negotiation.cc:407] Connection header received
0929 01:13:06.477388 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:06.477398 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:06.477470 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:06.477584 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:13:06.478283 (+ 699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36004: BlockingRecv error: recv got EOF from 127.0.0.1:36004 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 01:13:06.759969 28980 catalog_manager.cc:1259] Loaded cluster ID: 34f13e98b40d4e378641c535030c003f
I20260929 01:13:06.760092 28980 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b: loading cluster ID for follower catalog manager: success
I20260929 01:13:06.762888 28980 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b: acquiring CA information for follower catalog manager: success
I20260929 01:13:06.764739 28980 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60097088018443b6bc7a95c830f5a87b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:13:07.004777 29048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:06.998161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.5.198.253:43107)
0929 01:13:06.998978 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:06.998985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:06.999008 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:13:06.999090 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:06.999094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:06.999234 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:06.999376 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:13:06.999810 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:07.000893 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:07.002317 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:07.002820 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:07.002960 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:07.003495 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:13:07.004099 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:13:07.004281 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:07.004333 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 01:13:07.004420 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":533,"threads_started":1}
I20260929 01:13:07.017720 29054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:07.010937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44342 (local address 127.5.198.252:38157)
0929 01:13:07.011628 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:07.011633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:07.011649 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:13:07.011734 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:07.011738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:07.011876 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:07.012015 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:13:07.012549 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:07.013232 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:07.015204 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:07.015604 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:07.015698 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:07.016357 (+ 659us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:13:07.017057 (+ 700us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:13:07.017175 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:07.017302 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 01:13:07.017364 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":459,"threads_started":1}
I20260929 01:13:07.034610 29048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:07.023805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.5.198.253:43107)
0929 01:13:07.024046 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:07.024053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:07.025631 (+ 1578us) server_negotiation.cc:407] Connection header received
0929 01:13:07.025939 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:07.025948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:07.026070 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:07.026146 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:13:07.026982 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:07.028434 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:07.031243 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:07.032351 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:07.032576 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:07.033479 (+ 903us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:13:07.033676 (+ 197us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:13:07.033755 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:07.034025 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 01:13:07.034150 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 01:13:07.042094 29054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:07.024845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44354 (local address 127.5.198.252:38157)
0929 01:13:07.025030 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:07.025037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:07.035493 (+ 10456us) server_negotiation.cc:407] Connection header received
0929 01:13:07.035752 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:07.035759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:07.035844 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:07.035904 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:13:07.036521 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:07.037672 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:07.039737 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:07.040390 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:07.040517 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:07.041111 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:13:07.041380 (+ 269us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:13:07.041457 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:07.041713 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 01:13:07.041808 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:13:07.043565 29061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:13:07.024203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.5.198.254:34893)
0929 01:13:07.035745 (+ 11542us) server_negotiation.cc:206] Beginning negotiation
0929 01:13:07.035750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:13:07.035768 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:13:07.035874 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:13:07.035881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:13:07.036126 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:13:07.037091 (+ 965us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:13:07.037270 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:07.038270 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:07.040448 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:13:07.041084 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:13:07.041849 (+ 765us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:13:07.042035 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:13:07.042500 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:13:07.043167 (+ 667us) server_negotiation.cc:1035] Waiting for connection context
0929 01:13:07.043242 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 01:13:07.043310 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11405,"thread_start_us":12088,"threads_started":1}
I20260929 01:13:07.048574 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 28699
I20260929 01:13:07.049655 28761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:13:07.187849 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 28699
I20260929 01:13:07.208841 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 28773
I20260929 01:13:07.209926 28835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:13:07.315760 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 28773
I20260929 01:13:07.336568 5915 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu pid 28847
I20260929 01:13:07.337786 28909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:13:07.447196 5915 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskd3rvDY/build/asan/bin/kudu with pid 28847
2026-09-29T01:13:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4955 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9904 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (364122 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST